Active training method and device for upper limb rehabilitation robot and upper limb rehabilitation robot

Abstract

The present invention provides an active training method and device for an upper limb rehabilitation robot, and an upper limb rehabilitation robot; wherein the method comprises: extracting a region of interest (ROI) from an RGB image based on a forward kinematics model to obtain an ROI image; calculating the dense optical flow of the ROI image based on a Farneback optical flow algorithm, and determining the neighborhood of the end of an arm to be trained in the ROI image; determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood; finally determining the target velocity vector of the end of the arm to be trained, and determining the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector, and assisting the arm to be trained in rehabilitation training according to the movement intention until it moves to the target position; the above training method reduces the active training cost of the upper limb rehabilitation robot, improves the practicability of the active training mode, and is convenient for popularization and implementation in practical applications.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation training, and in particular to an active training method and device for an upper limb rehabilitation robot and an upper limb rehabilitation robot. Background Art

[0002] Cerebral stroke, commonly known as stroke, is an acute cerebrovascular circulatory disorder caused by blockage or rupture of cerebral blood vessels. It has the characteristics of high morbidity, high mortality, high disability rate and high recurrence rate. Among stroke patients, only a few mild patients can recover naturally. Most stroke patients will have residual motor, sensory, cognitive, speech and other functional disorders to varying degrees, which seriously affect their daily activities and quality of life. Rehabilitation is the most effective way to reduce the disability rate of stroke. Timely, scientific and effective rehabilitation training, especially in the "golden period" of the early stage of the disease, plays a vital role in treating the disease, restoring function, preventing stroke recurrence and reducing complications. In traditional stroke rehabilitation treatment, doctors mainly provide one-to-one rehabilitation treatment to patients in a bare-handed manner. The doctor's personal treatment methods, experience differences, subjective consciousness and fatigue level will directly affect the treatment effect. The treatment process is labor-intensive and the nursing cost is high. In addition, the ratio of doctors to patients is seriously unbalanced, which makes it difficult to meet the growing medical needs. Therefore, the introduction of medical rehabilitation robot equipment is a feasible solution to help effectively alleviate the contradiction between rehabilitation supply and demand.

[0003] Rehabilitation robots can assist or even replace doctors in providing patients with more continuous, effective and targeted rehabilitation training and treatment, alleviate the shortage of rehabilitation medical human resources, and can record patients' treatment data in real time, providing an objective basis for disease assessment and program improvement. In practical applications, rehabilitation robots mainly provide two rehabilitation training modes: active and passive. With the in-depth study of the theory and practice of neural plasticity and functional reorganization, the role of active rehabilitation is far greater than that of passive movement.

[0004] Active rehabilitation training of rehabilitation robots means that the patient actively initiates movement, but because the patient's limbs are damaged and cannot independently generate all the force / torque required to complete the movement, the rehabilitation robot is required to provide some auxiliary force / torque. After recognizing the patient's movement intention, the rehabilitation robot provides the required auxiliary force / torque to assist the patient in completing the movement. Among them, the difficulty of active rehabilitation training lies in how to identify the patient's movement intention. The existing methods mainly recognize the patient's movement intention through EEG or EMG signals to achieve active training, but the sensors for collecting EEG or EMG signals are relatively expensive and lack practicality, which greatly limits the promotion of the active training mode of rehabilitation robots. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide an upper limb rehabilitation robot active training method, device and upper limb rehabilitation robot to alleviate the above-mentioned problems, reduce the active training cost of the upper limb rehabilitation robot, improve the practicality of the active training mode, and facilitate promotion and implementation in practical applications.

[0006] In a first aspect, an embodiment of the present invention provides an active training method for an upper limb rehabilitation robot, the method comprising: acquiring an RGB image including an arm to be trained, and extracting a region of interest (ROI) from the RGB image based on a forward kinematics model to obtain an ROI image; wherein the ROI image covers all motion areas of the arm to be trained; calculating the dense optical flow of the ROI image based on a Farneback optical flow algorithm; acquiring the current position of the end of the arm to be trained in the ROI image, and determining the neighborhood of the end of the arm to be trained in the ROI image based on the current position; determining the actual velocity vector of the end of the arm to be trained based on the optical flow in the neighborhood; acquiring the target position of the end of the arm to be trained in the ROI image, and determining the target velocity vector of the end of the arm to be trained based on the target position and the current position; determining the movement intention of the arm to be trained based on the actual velocity vector and the target velocity vector, and assisting the arm to be trained in rehabilitation training based on the movement intention until it moves to the target position.

[0007] Preferably, the step of calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm includes: performing grayscale conversion processing on the ROI image to obtain a first grayscale image; performing median filtering processing on the first grayscale image to obtain a filtered second grayscale image; based on the Farneback optical flow algorithm, calculating the optical flow of the second grayscale image of the current frame according to the second grayscale image of the current frame and the second grayscale image of the previous frame.

[0008] Preferably, the step of determining the neighborhood of the end of the arm to be trained in the ROI image according to the current position includes: determining the neighborhood width and the neighborhood height according to the shape of the arm to be trained; and establishing a rectangular area with a width equal to the neighborhood width and a height equal to the neighborhood height with the current position as the center, as the neighborhood of the end of the arm to be trained in the ROI image.

[0009] Preferably, the optical flow includes a first optical flow in the X direction and a second optical flow in the Y direction; the step of determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood includes: averaging all the first optical flows in the neighborhood to obtain a first velocity in the X direction; and averaging all the second optical flows in the neighborhood to obtain a second velocity in the Y direction; and obtaining the actual velocity vector according to the first velocity and the second velocity.

[0010] Preferably, the step of determining the movement intention of the arm to be trained based on the actual velocity vector and the target velocity vector includes: calculating the projection of the actual velocity vector on the target velocity vector; judging whether the projection meets the preset movement condition; if so, determining that the arm to be trained has the correct movement intention, and assisting the arm to be trained in rehabilitation training until it moves to the target position.

[0011] Preferably, the step of determining whether the projection satisfies a preset motion condition comprises: determining whether the direction of the projection is positive, and whether the amplitude of the projection reaches a preset threshold; if both are true, determining that the projection satisfies the preset motion condition.

[0012] Preferably, the method further comprises: if the projection does not satisfy a preset movement condition, determining that the arm to be trained has an erroneous movement intention, and stopping assisting the arm to be trained in performing rehabilitation training.

[0013] In a second aspect, an embodiment of the present invention further provides an upper limb rehabilitation robot active training device, the device comprising: an image acquisition module, used to acquire an RGB image containing an arm to be trained, and extract a region of interest ROI from the RGB image based on a forward kinematics model to obtain an ROI image; wherein the ROI image covers all motion areas of the arm to be trained; an optical flow calculation module, used to calculate the dense optical flow of the ROI image based on the Farneback optical flow algorithm; a neighborhood determination module, used to acquire the current position of the end of the arm to be trained in the ROI image, and determine the neighborhood of the end of the arm to be trained in the ROI image according to the current position; an actual speed determination module, used to determine the actual speed vector of the end of the arm to be trained according to the optical flow in the neighborhood; a target speed determination module, used to acquire the target position of the end of the arm to be trained in the ROI image, and determine the target speed vector of the end of the arm to be trained according to the target position and the current position; a motion intention determination module, used to determine the motion intention of the arm to be trained according to the actual speed vector and the target speed vector, and assist the arm to be trained in rehabilitation training according to the motion intention until it moves to the target position.

[0014] In a third aspect, an embodiment of the present invention further provides an upper limb rehabilitation robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method of the first aspect are executed.

[0016] The embodiments of the present invention bring the following beneficial effects:

[0017] The embodiment of the present invention provides an active training method and device for an upper limb rehabilitation robot, and an upper limb rehabilitation robot. First, a region of interest (ROI) is extracted from an RGB image based on a forward kinematics model to obtain an ROI image; the dense optical flow of the ROI image is calculated based on a Farneback optical flow algorithm, and a neighborhood of the end of an arm to be trained in the ROI image is determined; the actual velocity vector of the end of the arm to be trained is determined according to the optical flow in the neighborhood; finally, the target velocity vector of the end of the arm to be trained is determined, and the movement intention of the arm to be trained is determined according to the actual velocity vector and the target velocity vector, and the arm to be trained is assisted in rehabilitation training according to the movement intention until it moves to the target position; the above-mentioned training method will cause a slight displacement of the limbs after the patient initiates active movement, and the size and direction of the displacement are captured by the RGB image to identify the patient's movement intention to realize active training, thereby reducing the active training cost of the upper limb rehabilitation robot, improving the practicality of the active training mode, and facilitating promotion and implementation in practical applications.

[0018] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description and the drawings.

[0019] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A schematic diagram of the structure of an upper limb rehabilitation robot provided by an embodiment of the present invention;

[0022] Figure 2 A flowchart of an active training method for an upper limb rehabilitation robot provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a forward kinematics model provided by an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a region of interest provided by an embodiment of the present invention;

[0025] Figure 5A schematic diagram of a neighborhood of an arm end to be trained provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of optical flow in a motion state of an arm end to be trained provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of an upper limb rehabilitation robot active training device provided by an embodiment of the present invention;

[0028] Figure 8 A schematic structural diagram of another upper limb rehabilitation robot provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] To facilitate understanding of this embodiment, the active training method for an upper limb rehabilitation robot provided by an embodiment of the present invention is first described in detail below. Figure 1 As shown, the upper limb rehabilitation robot includes a display terminal 10 and a host terminal 20; specifically, the display terminal 10 includes a display device 101 and a visual sensor 102, the display device 101 is used to display the ROI (Region Of Interest) image, the optical flow in the neighborhood of the end of the arm to be trained, and the end target position in real time, and the visual sensor 102 is fixedly installed on the display terminal 10, and is used to obtain an RGB image containing the arm to be trained; the host terminal 20 includes an upper limb exoskeleton 201 and a control host 202, the upper limb exoskeleton 201 is used to provide auxiliary force / torque to assist the patient to complete rehabilitation training, and the control host 202 is used to adjust the auxiliary force / torque output by the upper limb exoskeleton 201.

[0031] It should be noted that, in actual applications, the upper limb exoskeleton 201 has the function of switching between left and right hands, as well as the function of adjusting the length of the upper arm and the forearm, and includes at least the following four degrees of freedom: shoulder joint swing / adduction degree of freedom, flexion / extension degree of freedom, internal rotation / external rotation degree of freedom, and elbow joint flexion / extension degree of freedom, which can be specifically set according to actual conditions.

[0032] Based on the above-mentioned upper limb rehabilitation robot, an embodiment of the present invention provides an active training method for an upper limb rehabilitation robot, such as Figure 2 As shown, the method comprises the following steps:

[0033] Step S202, obtaining an RGB image containing the arm to be trained, and extracting a region of interest ROI from the RGB image based on a forward kinematics model to obtain an ROI image;

[0034] Specifically, when the patient (not completely paralyzed) initiates active movement, the arm to be trained will produce a slight displacement. At this time, the visual sensor can capture the size and direction of the displacement and generate an RGB image containing the arm to be trained. After the control host obtains the RGB image, it crops the RGB image based on the forward kinematics model to extract the region of interest ROI image, where the ROI image covers all movement areas of the arm to be trained.

[0035] For the forward kinematics model, such as Figure 3 As shown in FIG. 1 , the upper limb is modeled as a two-link structure (upper arm + lower arm) with four degrees of freedom, where the coordinate system XYZ represents the camera coordinate system; SH[x 0 ,y 0 ,z 0 ] is the position coordinate of the shoulder joint center in the camera coordinate system (determined by the installation position); EL represents the elbow joint center; WR represents the wrist joint center, that is, the end of the arm to be trained; L u is the length of the upper arm (the distance from SH to EL); L l is the length of the forearm (the distance from EL to WR); q1, q2, q3, and q4 represent the angles of shoulder swing / adduction, shoulder flexion / extension, shoulder internal rotation / external rotation, and elbow flexion / extension, respectively.

[0036] Based on the above forward kinematics model, the position of the end of the arm to be trained (i.e., the wrist joint center WR) in the camera coordinate system can be determined, namely:

[0037]

[0038] Then, by transforming the camera intrinsic parameter matrix K to the pixel coordinate system, we can obtain the pixel coordinates of the end of the arm to be trained in the RGB image:

[0039]

[0040] Among them, [x 0 ,y 0 ,z 0 ] represents the position coordinates of the shoulder joint center in the camera coordinate system (determined by the installation position), L u Indicates the length of the upper arm, L l represents the length of the forearm, q1, q2, q3, q4 represent the angles of shoulder joint external swing / adduction, shoulder joint flexion / extension, shoulder joint internal rotation / external rotation, and elbow joint flexion / extension, respectively. e ,y e] represents the pixel coordinates of the wrist joint center in the RGB image pixel coordinate system, K∈R 3×3 Represents the intrinsic parameter matrix of the camera (given by the manufacturer). The camera coordinate system and the pixel coordinate system are transformed through the intrinsic parameter matrix.

[0041] Therefore, with q1, q2, q3, q4, L u and L l As a constraint, we can establish an optimization problem to find x e The minimum value x e,min and the maximum value x e,max , and y e The minimum value y e,min and the maximum value y e,max , intercept [x e,min -w e :x e,max +w e ,y e,min -h e :y e,max +h e ] is the ROI area, such as Figure 4 The gray shaded area shown in the figure. The figure uses a standard pixel coordinate system, with the origin at the upper left corner of the image, the X axis pointing to the right, and the Y axis pointing downward; e,min is the minimum x-coordinate of the wrist joint center, x e,max is the maximum value of the x coordinate of the wrist joint center; y e,min is the minimum value of the y coordinate of the wrist joint center; y e,max is the maximum value of the y coordinate of the wrist joint center; w e Indicates the width margin, h e It should be noted that the width margin w e and height margin h e It can be set based on historical experience or experimental values.

[0042] In summary, by extracting the ROI image from the RGB image, the amount of calculation can be reduced, thereby improving the real-time performance of the algorithm. Since one of the disadvantages of the optical flow method is its slow operation speed, extracting the ROI image is equivalent to cropping the original RGB image output by the visual sensor. The cropping range is all possible motion areas of the arm to be trained. Subsequent algorithms are performed on the cropped image, namely the ROI image, thereby reducing the amount of calculation and improving the operation speed of the optical flow method.

[0043] Step S204, calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm;

[0044] At present, the commonly used visual motion detection algorithms mainly include frame difference method, background difference method and optical flow method. The optical flow method can detect independent moving targets without knowing any information of the scene in advance, so it can obtain complete information of the moving target, so it is suitable for dynamic backgrounds. In addition, compared with neural networks, neural networks are more suitable for identifying large displacement movements, and the effect of identifying small displacements is average. Therefore, for the small displacements of the limbs after the patient initiates active movement, the embodiment of the present invention uses the optical flow method for visual motion detection.

[0045] Among them, the traditional optical flow method mainly includes dense optical flow and sparse optical flow. Since dense optical flow is an image registration method for point-by-point matching of an image or a specified area, it calculates the offset of all points on the image to form a dense optical flow field. Through this dense optical flow field, pixel-level image registration can be performed. Therefore, the embodiment of the present invention calculates the dense optical flow of the ROI image. Commonly used methods for calculating dense optical flow mainly include Brox algorithm, Farneback algorithm and TVL1 algorithm. Since the Farneback algorithm has good real-time performance and the frame rate on a mid-to-low-end GPU (Graphics Processing Unit, graphics processor) can reach 30 frames, it can meet the needs of real-time use. Therefore, the embodiment of the present invention uses the Farneback optical flow algorithm to calculate the dense optical flow of the ROI image.

[0046] Specifically, for the above ROI image, the process of calculating the optical flow is as follows: grayscale conversion is performed on the ROI image to obtain the first grayscale image ROI_gray; median filtering is performed on the first grayscale image ROI_gray to obtain the filtered second grayscale image ROI_gray_filted; based on the Farneback optical flow algorithm, the optical flow flow of the second grayscale image of the current frame is calculated according to the second grayscale image ROI_gray_filted of the current frame and the second grayscale image ROI_gray_filted of the previous frame. Among them, the optical flow flow consists of the optical flow in the X and Y directions, which is further split into the optical flow flowx in the X direction and the optical flow flowy in the Y direction, and W=x e,max -x e,min +2w e , H=y e,max -y e,min +2h e , then flowx and flowy are both matrices of dimension W×H.

[0047] Step S206, obtaining the current position of the end of the arm to be trained in the ROI image, and determining the neighborhood of the end of the arm to be trained in the ROI image according to the current position;

[0048] Since the ROI image is obtained by cropping the original RGB image, the pixel coordinates of the end of the arm to be trained in the ROI image are offset from the pixel coordinates in the RGB image. The pixel coordinates of the end of the arm to be trained (i.e., the center of the wrist joint) in the ROI image can be determined according to the following formula:

[0049]

[0050] Among them, [x e ,y e ] represents the pixel coordinates of the wrist joint center in the RGB image pixel coordinate system, w e Indicates the width margin, h e Indicates the height margin, Indicates the pixel coordinates of the end of the arm to be trained in the ROI image, that is, the current position.

[0051] like Figure 5 As shown, the figure adopts a standard pixel coordinate system, with the origin at the upper left corner of the image, the X axis pointing to the right, and the Y axis pointing downward; Indicates the x-coordinate of the end of the arm to be trained in the pixel coordinate system of the ROI image. Indicates the y coordinate of the end of the arm to be trained in the pixel coordinate system of the ROI image; represents the neighborhood width, represents the neighborhood height, and It can be determined based on the shape of the arm, based on the current position of the end of the arm to be trained in the ROI image The process of determining the neighborhood is as follows: Determine the neighborhood width based on the shape of the arm to be trained and neighborhood height By current location As the center, establish the width as the neighborhood width Height is the neighborhood height The rectangular area is used as the neighborhood of the end of the arm to be trained in the ROI image. Figure 5 It should be noted that when it is in a stationary state as shown in the figure, the optical flow of each pixel in the neighborhood is approximately zero, and it looks like a point.

[0052] Step S208, determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood;

[0053] For the optical flow in the motion state of the end of the arm to be trained, such as Figure 6 As shown, [V e,x ,V e,y ] represents the actual velocity vector of the end of the arm to be trained, represents the target position of the upper limb end, [V d,x ,Vd,y ] represents the target motion speed of the end of the arm to be trained, i.e., the target velocity vector. When the arm to be trained makes a small movement to the left, some pixels in the neighborhood generate a component in the positive direction of X, which looks like an arrow pointing right (due to the mirror image, the direction is opposite to the actual movement). Since the optical flow includes the first optical flow in the X direction, i.e., flowx, and the second optical flow in the Y direction, i.e., flowy; therefore, the actual velocity vector [V] of the end of the arm to be trained can be obtained by averaging the optical flow in the neighborhood of the end of the arm to be trained. e,x ,V e,y ].

[0054] Specifically, the process of determining the actual velocity vector of the end of the arm to be trained based on the optical flow in the neighborhood is as follows: All the first optical flows in the neighborhood are averaged to obtain the first velocity in the X direction, i.e., V e,x ; and, average all the second optical flows in the neighborhood to obtain the second velocity in the Y direction, i.e., V e,y ; Obtain the actual velocity vector based on the first velocity and the second velocity.

[0055] The average calculation formula is as follows:

[0056]

[0057] Among them, [i,j]∈S represents all pixel coordinates in the field S, flowx[i,j] represents the element in the i-th row and j-th column of the matrix flowx, and flowy[i,j] represents the element in the i-th row and j-th column of the matrix flowy. represents the neighborhood width, Represents the neighborhood height.

[0058] The above method obtains the actual velocity vector of the end of the arm to be trained by averaging the optical flow in the neighborhood of the end of the arm to be trained, rather than the optical flow of a single pixel at the end of the arm to be trained, thereby avoiding the influence of noise on the optical flow of a single pixel, which causes the obtained velocity vector to fail to reflect the true movement intention. Therefore, by averaging the optical flow in the neighborhood of the end of the arm to be trained, the influence of noise can be reduced, and the calculation accuracy of the actual velocity vector can be improved.

[0059] Step S210, obtaining the target position of the end of the arm to be trained in the ROI image, and determining the target velocity vector of the end of the arm to be trained according to the target position and the current position;

[0060] Specifically, the target velocity vector [V d,x ,V d,y ]:

[0061]

[0062] in, Indicates the target position of the end of the arm to be trained in the ROI image, that is, the target position of the end of the upper limb, such as Figure 5 Or as shown in 6, represents the current position, and T represents the preset motion duration. It should be noted that the target position The position corresponding to the current training task in the goal-oriented rehabilitation training provided for the upper limb rehabilitation robot. When the current training task is completed, the control host generates the target position corresponding to the next training task until all tasks are completed or the training time ends.

[0063] Step S212, determining the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector, and assisting the arm to be trained in performing rehabilitation training according to the movement intention until the arm moves to the target position.

[0064] Specifically, the process of determining the movement intention of the arm to be trained based on the actual velocity vector and the target velocity vector includes: calculating the projection of the actual velocity vector on the target velocity vector; judging whether the projection meets the preset movement condition; if so, determining that the arm to be trained has the correct movement intention, and assisting the arm to be trained in rehabilitation training until it moves to the target position.

[0065] The projection V of the actual velocity vector on the target velocity vector is calculated according to the following formula:

[0066]

[0067] Among them, [V d,x ,V d,y ] represents the target velocity vector, [V e,x ,V e,y ] represents the actual velocity vector.

[0068] For the above projection V, the process of determining whether the projection meets the preset motion condition includes: determining whether the direction of the projection is positive, and whether the amplitude of the projection reaches the preset threshold; if both are true, determining that the projection meets the preset motion condition. That is, when the direction of the projection is positive and the amplitude reaches the preset threshold, it is determined that the arm to be trained has generated a motion intention consistent with the target direction, that is, a correct motion intention. At this time, the control host controls the exoskeleton to provide auxiliary force / torque to make the arm to be trained move at the target velocity vector [V d,x ,V d,y ]Move to the target position When the target position is reached, the current task is considered completed, and the control host generates the next target position until all tasks are completed or the training time ends.

[0069] Furthermore, the method also includes: if the projection does not meet the preset movement conditions, it is determined that the arm to be trained has an erroneous movement intention. At this time, the control host controls the exoskeleton to remain still, that is, stops assisting the arm to be trained in rehabilitation training until the arm to be trained has a correct movement intention.

[0070] The active training method for an upper limb rehabilitation robot provided in an embodiment of the present invention produces a slight displacement of the limbs after the patient initiates active movement. The size and direction of the displacement are captured by RGB images to identify the patient's movement intention to implement active training, thereby reducing the active training cost of the upper limb rehabilitation robot, improving the practicality of the active training mode, and facilitating promotion and implementation in practical applications.

[0071] In summary, the above-mentioned active training method for upper limb rehabilitation robots has the following advantages: ① The active movement intention of the patient's upper limbs can be identified based on an ordinary RGB camera. Compared with the method using EEG or EMG signals, it can greatly reduce the price of the required sensors, which is helpful to reduce the cost of active training of upper limb rehabilitation robots and improve the practicality of the active training mode; ② By performing straight-through filtering on the RGB image output by the visual sensor and cropping the region of interest (ROI) from the RGB image output by the visual sensor based on the positive kinematics model, the subsequent algorithms are all performed on the ROI image, which can significantly reduce the amount of calculation and improve the practicality of the optical flow method; ③ By averaging the optical flow in the neighborhood of the end of the arm to be trained, the influence of noise can be reduced, the accuracy of upper limb active movement intention recognition is improved, and it is easy to promote and implement in practical applications.

[0072] Corresponding to the above method embodiment, the embodiment of the present invention also provides an upper limb rehabilitation robot active training device, such as Figure 7 As shown, the device includes: an image acquisition module 71, an optical flow calculation module 72, a neighborhood determination module 73, an actual speed determination module 74, a target speed determination module 75 and a motion intention determination module 76; wherein the functions of each module are as follows:

[0073] The image acquisition module 71 is used to acquire an RGB image containing the arm to be trained, and extract a region of interest ROI from the RGB image based on a forward kinematics model to obtain an ROI image; wherein the ROI image covers all motion areas of the arm to be trained;

[0074] An optical flow calculation module 72, used for calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm;

[0075] A neighborhood determination module 73 is used to obtain the current position of the end of the arm to be trained in the ROI image, and determine the neighborhood of the end of the arm to be trained in the ROI image according to the current position;

[0076] An actual speed determination module 74, used to determine the actual speed vector of the end of the arm to be trained according to the optical flow in the neighborhood;

[0077] A target speed determination module 75 is used to obtain the target position of the end of the arm to be trained in the ROI image, and determine the target speed vector of the end of the arm to be trained according to the target position and the current position;

[0078] The movement intention determination module 76 is used to determine the movement intention of the arm to be trained according to the actual speed vector and the target speed vector, and assist the arm to be trained in rehabilitation training according to the movement intention until it moves to the target position.

[0079] The active training device for an upper limb rehabilitation robot provided in an embodiment of the present invention can generate a small displacement of the limbs after the patient initiates active movement. The size and direction of the displacement are captured by RGB images to identify the patient's movement intention to implement active training, thereby reducing the active training cost of the upper limb rehabilitation robot, improving the practicality of the active training mode, and facilitating promotion and implementation in practical applications.

[0080] Preferably, the above-mentioned optical flow calculation module 72 is also used for: performing grayscale conversion processing on the ROI image to obtain a first grayscale image; performing median filtering processing on the first grayscale image to obtain a filtered second grayscale image; based on the Farneback optical flow algorithm, according to the second grayscale image of the current frame and the second grayscale image of the previous frame, calculating the optical flow of the second grayscale image of the current frame.

[0081] Preferably, the above-mentioned neighborhood determination module 73 is also used to: determine the neighborhood width and neighborhood height according to the shape of the arm to be trained; and establish a rectangular area with a width equal to the neighborhood width and a height equal to the neighborhood height with the current position as the center, as the neighborhood of the end of the arm to be trained in the ROI image.

[0082] Preferably, the optical flow includes a first optical flow in the X direction and a second optical flow in the Y direction; the above-mentioned actual speed determination module 74 is also used to: average all the first optical flows in the neighborhood to obtain the first speed in the X direction; and average all the second optical flows in the neighborhood to obtain the second speed in the Y direction; and obtain the actual speed vector according to the first speed and the second speed.

[0083] Preferably, the above-mentioned movement intention determination module 76 is also used to: calculate the projection of the actual velocity vector on the target velocity vector; determine whether the projection meets the preset movement conditions; if so, determine that the arm to be trained has the correct movement intention, and assist the arm to be trained in rehabilitation training until it moves to the target position.

[0084] Preferably, determining whether the projection satisfies a preset motion condition comprises: determining whether the direction of the projection is positive, and whether the amplitude of the projection reaches a preset threshold; if both are true, determining that the projection satisfies the preset motion condition.

[0085] Preferably, the device further comprises: if the projection does not satisfy the preset movement condition, determining that the arm to be trained has an erroneous movement intention, and stopping assisting the arm to be trained in performing rehabilitation training.

[0086] The upper limb rehabilitation robot active training device provided by the embodiment of the present invention has the same technical features as the upper limb rehabilitation robot active training method provided by the above embodiment, so it can also solve the same technical problems and achieve the same technical effects.

[0087] An embodiment of the present invention further provides an upper limb rehabilitation robot, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned upper limb rehabilitation robot active training method.

[0088] See also Figure 8 As shown, the upper limb rehabilitation robot includes a processor 100 and a memory 101. The memory 101 stores machine executable instructions that can be executed by the processor 100. The processor 100 executes the machine executable instructions to implement the above-mentioned upper limb rehabilitation robot active training method.

[0089] Further, Figure 8 The upper limb rehabilitation robot shown further includes a bus 102 and a communication interface 103 , and the processor 100 , the communication interface 103 and the memory 101 are connected via the bus 102 .

[0090] Among them, the memory 101 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which can be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. can be used. The bus 102 can be an ISA (Industrial Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Enhanced Industry Standard Architecture) bus, etc. The above-mentioned bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 8Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0091] The processor 100 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 100. The above processor 100 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and completes the steps of the method of the above embodiment in combination with its hardware.

[0092] This embodiment also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned upper limb rehabilitation robot active training method.

[0093] The active training method and device for an upper limb rehabilitation robot and the computer program product for an upper limb rehabilitation robot provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be found in the method embodiments and will not be repeated here.

[0094] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0095] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0097] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0098] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An active training method for an upper limb rehabilitation robot, characterized in that: The method comprises: Acquire an RGB image containing the arm to be trained, and extract a region of interest ROI from the RGB image based on a forward kinematics model to obtain a ROI image; wherein the ROI image covers all motion areas of the arm to be trained; Calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm; Acquire the current position of the end of the arm to be trained in the ROI image, and determine the neighborhood of the end of the arm to be trained in the ROI image according to the current position; wherein the neighborhood width and neighborhood height are determined according to the shape of the arm to be trained; and establish a rectangular area with a width equal to the neighborhood width and a height equal to the neighborhood height with the current position as the center, as the neighborhood of the end of the arm to be trained in the ROI image; Determine the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood; Acquire the target position of the end of the arm to be trained in the ROI image, and determine the target velocity vector of the end of the arm to be trained according to the target position and the current position; The movement intention of the arm to be trained is determined according to the actual velocity vector and the target velocity vector, and the arm to be trained is assisted in performing rehabilitation training according to the movement intention until the arm moves to the target position.

2. The method according to claim 1, characterized in that The step of calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm comprises: Performing grayscale conversion processing on the ROI image to obtain a first grayscale image; Performing median filtering on the first grayscale image to obtain a filtered second grayscale image; Based on the Farneback optical flow algorithm, the optical flow of the second grayscale image of the current frame is calculated according to the second grayscale image of the current frame and the second grayscale image of the previous frame.

3. The method according to claim 1, characterized in that The optical flow includes a first optical flow in the X direction and a second optical flow in the Y direction; the step of determining the actual velocity vector of the end of the arm to be trained according to the optical flow in the neighborhood includes: averaging all the first optical flows in the neighborhood to obtain a first velocity in the X direction; and averaging all the second optical flows in the neighborhood to obtain a second velocity in the Y direction; The actual speed vector is obtained according to the first speed and the second speed.

4. The method according to claim 1, characterized in that: The step of determining the movement intention of the arm to be trained according to the actual velocity vector and the target velocity vector comprises: Calculating the projection of the actual velocity vector on the target velocity vector; Determining whether the projection satisfies a preset motion condition; If yes, it is determined that the arm to be trained has the correct movement intention, and the arm to be trained is assisted in rehabilitation training until it moves to the target position.

5. The method according to claim 4, characterized in that The step of determining whether the projection meets a preset motion condition comprises: Determine whether the direction of the projection is positive, and whether the amplitude of the projection reaches a preset threshold; If yes, it is determined that the projection meets the preset motion condition.

6. The method according to claim 4, characterized in that The method further comprises: If the projection does not satisfy the preset movement condition, it is determined that the arm to be trained has an erroneous movement intention, and assisting the arm to be trained in performing rehabilitation training is stopped.

7. An upper limb rehabilitation robot active training device, characterized in that: The device comprises: An image acquisition module, used to acquire an RGB image containing the arm to be trained, and extract a region of interest ROI from the RGB image based on a forward kinematics model to obtain an ROI image; wherein the ROI image covers all motion areas of the arm to be trained; An optical flow calculation module, used for calculating the dense optical flow of the ROI image based on the Farneback optical flow algorithm; A neighborhood determination module is used to obtain the current position of the end of the arm to be trained in the ROI image, and determine the neighborhood of the end of the arm to be trained in the ROI image according to the current position; wherein the neighborhood width and neighborhood height are determined according to the shape of the arm to be trained; and a rectangular area with a width of the neighborhood width and a height of the neighborhood height is established with the current position as the center, as the neighborhood of the end of the arm to be trained in the ROI image; An actual speed determination module, used to determine the actual speed vector of the end of the arm to be trained according to the optical flow in the neighborhood; A target speed determination module, used for acquiring the target position of the end of the arm to be trained in the ROI image, and determining the target speed vector of the end of the arm to be trained according to the target position and the current position; The movement intention determination module is used to determine the movement intention of the arm to be trained according to the actual speed vector and the target speed vector, and assist the arm to be trained in rehabilitation training according to the movement intention until it moves to the target position.

8. An upper limb rehabilitation robot, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are executed.

Images