Upper limb rehabilitation robot control system based on multi-modal data

Abstract

The application provides an upper limb rehabilitation robot control system based on multi-modal data, which comprises a first acquisition module, a second acquisition module and a third acquisition module. The first acquisition module collects torque signals corresponding to each joint of the rehabilitation robot to obtain a torque signal set. The second acquisition module acquires implicit neurons and original muscle force data of the upper limb end of a patient, and acquires a muscle force action signal set of the upper limb end of the patient according to the implicit neurons and the original muscle force data. The third acquisition module collects eye movement data information of the patient, compensates the position of the eye movement data information, and obtains an eye movement recognition signal set. A solving module establishes a multi-modal state vector according to the torque signal set, the muscle force action signal set and the eye movement recognition signal set, establishes a nonlinear state transition equation according to the multi-modal state vector and solves the nonlinear state transition equation, and predicts the movement intention of the patient according to the result. A control module controls the movement trajectory of the rehabilitation robot according to the movement intention of the patient. The control method provided by the application has good rehabilitation effect and is suitable for wide range of promotion.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of medical devices, in particular to a control system of an upper limb rehabilitation robot based on multi-modal data. BACKGROUND

[0002] The upper limb rehabilitation training mainly trains the functions of the shoulder, elbow, wrist, metacarpophalangeal and fingers, and according to the severity of the patient, the upper limb rehabilitation training includes active training and passive power-assisted training under the guidance of a rehabilitation training instructor.

[0003] Generally, the active training is adopted for the slight patients, and the patients perform the corresponding rehabilitation training according to subjective consciousness, the passive power-assisted training is adopted for the severe patients, and the current common passive power-assisted training mainly includes nurse interactive training and robot-assisted training, the nurse interactive training has a high requirement for the experience of the nurses and is difficult to popularize, the robot-assisted training has a preset moving track of the robot and lacks interaction with the patients, and it is difficult to meet the actual needs of the patients; how to predict the movement intention of the patient at the next moment according to the current movement situation of the patient, control the moving track of the rehabilitation robot according to the movement intention of the patient, and improve the rehabilitation effect of the patient gradually becomes a hot issue of the assisted rehabilitation. SUMMARY

[0004] Therefore, the present application aims to provide an upper limb rehabilitation robot control method and system based on multi-modal data to solve the technical problems in the prior art.

[0005] The present application provides an upper limb rehabilitation robot control method based on multi-modal data, which comprises the following steps:

[0006] Collecting the human-machine interaction force corresponding to each joint based on the torque sensors arranged at the joints of the upper limb rehabilitation robot, obtaining the torque signals corresponding to the human-machine interaction force of each joint according to the torque Jacobian matrix, and obtaining a torque signal set;

[0007] Obtaining the implicit neurons of the upper limb end of the patient, collecting the original muscle force data based on the muscle force sensors arranged at the upper limb end of the patient, and obtaining a muscle force action signal set of the upper limb end of the patient according to the implicit neurons and the original muscle force data;

[0008] Collecting the eye movement data information of the patient according to the head-wearing sensor of the patient, performing position compensation on the eye movement data information to eliminate the boundary points in the eye movement data information, and obtaining an eye movement recognition signal set;

[0009] Establishing a multi-modal state vector according to the torque signal set, the end muscle force action signal set and the eye movement recognition signal set, establishing a nonlinear state transition equation according to the multi-modal state vector and solving the nonlinear state transition equation, and predicting the movement intention of the patient according to the solving result;

[0010] Controlling the motion trajectory of the rehabilitation robot according to the predicted patient motion intention.

[0011] Preferably, the expression of the torque signal set is:

[0012]

[0013] In the formula, denotes the torque signal set at time t, denotes the joint torque signal at time t, denotes the number of joints;

[0014]

[0015] In the formula, denotes the torque Jacobian matrix, denotes the joint human-robot interaction force collected by the torque sensor at time t.

[0016] Preferably, the expression of the muscle force action signal set is:

[0017]

[0018] In the formula, denotes the muscle force action signal set at time t, denotes the number of muscle force sensors arranged at the end of the upper limb, denotes the number of hidden neurons, denotes the weight vector between the hidden layer neurons and the output neurons, denotes the original muscle force data;

[0019]

[0020] In the formula, denotes the muscle force sensor original muscle force data collected at time t.

[0021] Preferably, the step of collecting eye movement data information of the patient according to the sensor worn on the head of the patient, positionally compensating the eye movement data information to eliminate boundary points in the eye movement data information, and obtaining an eye movement recognition signal set is:

[0022] Collecting eye movement data information of the patient according to the sensor worn on the head of the patient, setting a neighborhood of the fixation point in the eye movement data information, and determining a core object in the fixation point;

[0023] Judging the density attribute of the fixation point based on the determined core object to eliminate boundary points in the eye movement data information.

[0024] According to the density attribute relationship of the gaze points, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

[0025] Preferably, the neighborhood of the gaze point is expressed as:

[0026]

[0027] Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold;

[0028] If the gaze point Neighborhood satisfy:

[0029]

[0030] The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations;

[0031] The expression of the eye movement recognition signal set is:

[0032]

[0033] Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations;

[0034]

[0035] Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

[0036] Preferably, the expression of the multimodal state vector is:

[0037]

[0038] Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. respectively represent the derivative of the torque signal set, the muscle force action signal set and the eye movement recognition signal set with respect to time at time t;

[0039] The expression of the nonlinear state transition equation is:

[0040]

[0041] In the formula, represents the multimodal state vector of the rehabilitation robot at time t+1, represents a nonlinear state transition function, represents a preset instruction control vector of external input, represents a control process noise.

[0042] Preferably, the step of establishing a nonlinear state transition equation according to the multimodal state vector and solving, and predicting the movement intention of the patient according to the solving result is:

[0043] Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established;

[0044] According to the state prediction equation corresponding to the nonlinear evolution relationship, a state transition Jacobian matrix is obtained, and a state covariance prediction equation is established according to the state transition Jacobian matrix;

[0045] According to the state prediction equation corresponding to the nonlinear evolution relationship, an observation prediction equation and an observation Jacobian matrix are established;

[0046] According to the state covariance prediction equation and the observation Jacobian matrix, a Kalman gain value of the nonlinear state transition equation is calculated;

[0047] Based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 and the observation prediction equation, a state prediction equation update solution is solved, and the movement intention of the patient is further predicted.

[0048] Preferably, the expression of the state prediction equation corresponding to the nonlinear evolution relationship is:

[0049]

[0050] In the formula, represents a state vector at time t+1 predicted based on information at time t, represents a nonlinear state transition function, represents t a state optimal state estimation value after filtering and correction at time t;

[0051] The expression of the state transition Jacobian matrix is:

[0052]

[0053] In the formula, represents the state transition Jacobian matrix;

[0054] The expression of the state covariance prediction equation is:

[0055]

[0056] In the formula, represents the t+1 time covariance predicted based on the t time information, represents the t time filtered and corrected state optimal state covariance, represents the rehabilitation robot compensation value;

[0057] The expression of the observation prediction equation is:

[0058]

[0059] In the formula, represents the t+1 time observation value predicted based on the t time information, represents the observation function;

[0060] The expression of the observation Jacobian matrix is:

[0061]

[0062] In the formula, represents the observation Jacobian matrix, representing the linearized gradient of the observation function at the predicted state point, used to calculate the sensitivity of the state to the observation;

[0063] The expression of the Kalman gain value is:

[0064]

[0065] In the formula, represents the t+1 time nonlinear state transition equation Kalman gain value, represents the observation noise covariance matrix, used to describe the sensor measurement error;

[0066] The expression of the state prediction equation update is:

[0067]

[0068] In the formula, represents the t+1 time filtered and corrected state optimal state estimation value, representing the patient's motion intention position, represents the t time corrected state optimal observation value.

[0069] Preferably, the upper limb rehabilitation robot control method based on multi-modal data further comprises: obtaining an updated covariance of a state prediction equation, and an expression of the updated covariance of the state prediction equation is:

[0070]

[0071] wherein, denotes a unit matrix.

[0072] The application further provides an upper limb rehabilitation robot control system based on multi-modal data, comprising:

[0073] A first acquisition module is configured to acquire human-machine interaction forces corresponding to joints based on torque sensors arranged at the joints of the upper limb rehabilitation robot, acquire torque signals corresponding to the human-machine interaction forces of the joints according to a torque Jacobian matrix, and obtain a torque signal set.

[0074] A second acquisition module is configured to acquire implicit neurons of an upper limb end of a patient, acquire original muscle force data based on muscle force sensors arranged at the upper limb end of the patient, and acquire a muscle force motion signal set of the upper limb end of the patient according to the implicit neurons and the original muscle force data.

[0075] A third acquisition module is configured to acquire eye movement data information of the patient according to a head-wearing sensor worn on the head of the patient, perform position compensation on the eye movement data information to eliminate boundary points in the eye movement data information, and obtain an eye movement recognition signal set.

[0076] A solving module is configured to establish a multi-modal state vector according to the torque signal set, the muscle force motion signal set of the upper limb end of the patient, and the eye movement recognition signal set, establish a nonlinear state transition equation according to the multi-modal state vector and solve the nonlinear state transition equation, and predict a movement intention of the patient according to a result of the solving.

[0077] A control module is configured to control a movement trajectory of the rehabilitation robot according to the predicted movement intention of the patient.

[0078] Compared with the prior art, the upper limb rehabilitation robot control method based on multi-modal data has the following beneficial effects: the torque sensors are arranged at the joints of the robot, the muscle force sensors are arranged at the upper limb end of the patient, and the head-wearing sensor is arranged in a head-wearing device worn on the head of the patient; the torque signal set, the muscle force motion signal set of the upper limb end of the patient, and the eye movement recognition signal set are acquired through the torque sensors, the muscle force sensors, and the head-wearing sensor respectively; the prediction model of the movement intention of the patient is established based on the acquired multi-modal data, the movement intention of the patient at the next moment is predicted, and the movement trajectory of the upper limb rehabilitation robot is controlled according to the movement intention of the patient, so that the effect of the rehabilitation robot in assisting the patient in rehabilitation is effectively improved, and the method is suitable for wide promotion.

[0079] Additional aspects and advantages of the present application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 A flow chart of the upper limb rehabilitation robot control method based on multi-modal data in the embodiment one of the present application;

[0081] Figure 2 A signal acquisition schematic diagram in the embodiment one of the present application;

[0082] Figure 3 A structure block diagram of the computer in the embodiment four of the present application.

[0083] The following detailed description will further describe the present application with reference to the above drawings. DETAILED DESCRIPTION

[0084] In order to facilitate the understanding of the present application, the present application will be described more fully below with reference to the relevant drawings. The drawings show several embodiments of the present application. However, the present application can be realized in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present application more thorough and comprehensive.

[0085] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein only for the purpose of describing specific embodiments and is not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more of the associated listed items.

[0086] Embodiment one

[0087] Please refer to Figure 1 , which shows the upper limb rehabilitation robot control method based on multi-modal data in the embodiment one of the present application. Specifically, the upper limb rehabilitation robot control method based on multi-modal data specifically includes steps S10 to S50:

[0088] S10, collecting the human-machine interaction force corresponding to each joint based on the torque sensor arranged at the joints of the upper limb rehabilitation robot, obtaining the torque signal corresponding to the human-machine interaction force of each joint according to the torque Jacobian matrix, and obtaining a torque signal set;

[0089] In specific implementation, as the patient's upper limb motor function gradually recovers, the patient needs to actively participate in the training process during the upper limb rehabilitation training. Therefore, human-computer interaction training should be carried out according to the patient's movement intention, which requires the control process of the upper limb rehabilitation robot to judge the patient's movement intention in advance; Figure 2 As shown, patients need to collect multimodal signals during the upper limb rehabilitation process. The torque sensor is arranged at the joints of the upper limb rehabilitation robot to obtain torque signal sets at different times; the muscle force sensor is arranged at the end of the patient's upper limb to obtain the patient's muscle force signal, and the head-worn sensor is arranged in the device worn by the patient to obtain scanning data when the patient turns his head.

[0090] The expression of the torque signal set is:

[0091]

[0092] Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints;

[0093]

[0094] Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

[0095] S20, obtaining hidden neurons at the end of the patient's upper limb, collecting raw muscle force data based on a muscle force sensor arranged at the end of the patient's upper limb, and obtaining a muscle force action signal set at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data;

[0096] Optionally, based on the contact status between the upper limb rehabilitation robot and the patient's upper limb extremities, the end muscle force signal can be collected to quickly and efficiently identify the patient's movement intention.

[0097] Optionally, the expression of the muscle action signal set is:

[0098]

[0099] Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, represents original muscle strength data;

[0100]

[0101] In the formula, represents a muscle strength sensor The original muscle strength data collected at time t.

[0102] S30, collecting eye movement data information of the patient according to the sensor worn on the head of the patient, performing position compensation on the eye movement data information to eliminate boundary points in the eye movement data information, and obtaining an eye movement recognition signal set;

[0103] Optionally, the step of collecting eye movement data information of the patient according to the sensor worn on the head of the patient, performing position compensation on the eye movement data information to eliminate boundary points in the eye movement data information, and obtaining an eye movement recognition signal set is:

[0104] Collecting eye movement data information of the patient according to the sensor worn on the head of the patient, setting a neighborhood of fixation points in the eye movement data information, and determining core objects in the fixation points;

[0105] Judging a density attribute of the fixation points based on the determined core objects to eliminate boundary points in the eye movement data information;

[0106] Eliminating boundary points in the eye movement data information according to the density attribute relationship of the fixation points, and deriving a density connected sample set as the eye movement recognition signal set;

[0107] Illustratively, the specific steps of judging the density attribute of the fixation points are:

[0108] If the fixation point A is located in the neighborhood of the fixation point B, and the fixation point B is a core object, it is said that A is directly covered by B in a dense area, that is, A is directly covered by B in a dense area.

[0109] If there is a string of fixation points B→C→D→…→A, each of which can directly reach the next point in density, such as B directly reaching C, C directly reaching D, and the last point directly reaching A, it is said that A is density reachable from B; that is, A indirectly belongs to the dense area where B is located through a series of core objects.

[0110] If there is a core object O, such that the fixation point A and the fixation point B are both density reachable from O, it is said that A and B are density connected; that is, A and B belong to the same dense area.

[0111] According to the density attribute relationship of the fixation points, boundary points in the eye movement data information are removed, and a density-continuous sample set is derived as an eye movement recognition signal set; effective eye movement signals are extracted, and through the density relationship analysis, points that are neither core objects nor can be associated to any core objects through density reachable relationship are eye movement errors or interference, and are considered as boundary points to be removed, and removing the boundary points in the eye movement data information is beneficial to solve the problems of displacement drift and the like caused by eye saccade in the signal collection process; and the remaining fixation point set after removing the boundary points is the eye movement recognition signal set.

[0112] The expression of the neighborhood of the fixation point is:

[0113]

[0114] In the formula, denotes a specified domain, denotes the distance between the fixation point and the fixation point at the time t, denotes a distance threshold value;

[0115] If the neighborhood of the fixation point satisfies:

[0116]

[0117] the fixation point is a core object, wherein, denotes a neighborhood fixation point quantity threshold value;

[0118] The expression of the eye movement recognition signal set is:

[0119]

[0120] In the formula, denotes the eye movement recognition signal set at the time t, denotes the eye movement recognition signal corresponding to the fixation point , and denotes the fixation point quantity.

[0121]

[0122] In the formula, denotes an initial eye movement recognition signal, denotes the eye movement data coefficient corresponding to the fixation point .

[0123] S40, establishing a multi-modal state vector according to the torque signal set, the end muscle force action signal set and the eye movement recognition signal set, establishing a non-linear state transition equation according to the multi-modal state vector and solving, and predicting the movement intention of the patient according to the solving result;

[0124] Optionally, in order to better adapt to the movement of the upper limb hemiplegic patient and improve the human-computer interaction performance of the upper limb rehabilitation robot, a high-efficiency movement intention recognition algorithm is designed based on the joint torque signal set, the muscle force action signal set and the eye movement recognition signal set collected by the above-mentioned sensors and the corresponding processing of the collected information, and then the movement intention of the patient is judged. In the embodiment, the joint torque signal, the muscle force action signal and the eye movement recognition signal are respectively:

[0125]

[0126]

[0127]

[0128] According to the above-mentioned collected signal data, the multi-modal data information of the torque sensor, the muscle force sensor and the head-wearing sensor can be obtained after corresponding processing. The multi-modal state vectors of the above-mentioned collected data information are established. The expression of the multi-modal state vector is:

[0129]

[0130] In the formula, X (t) represents the multi-modal state vector of the rehabilitation robot at t moment, respectively represent the torque signal set, the muscle force action signal set and the eye movement recognition signal set at t moment, respectively represent the derivative of the torque signal set, the muscle force action signal set and the eye movement recognition signal set with respect to time at t moment;

[0131] Considering the movement trajectory characteristics of the upper limb rehabilitation robot, the non-linear state transition equation is established based on the extended Kalman filter (EKF) and the expression of the non-linear state transition equation is:

[0132]

[0133] In the formula, X (t+1) represents the multi-modal state vector of the rehabilitation robot at t+1 moment, which is used to predict the movement intention of the patient at the next moment, f (X (t), u (t) ) represents the non-linear state transition function, u (t) represents the preset instruction control vector inputted from outside, w (t) represents the control process noise.

[0134] Optionally, the step of establishing a nonlinear state transition equation according to the multi-modal state vector and solving, predicting the patient's movement intention according to the solving result, is:

[0135] Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established;

[0136] According to the state prediction equation corresponding to the nonlinear evolution relationship, a state transition Jacobian matrix is obtained, and a state covariance prediction equation is established according to the state transition Jacobian matrix;

[0137] According to the state prediction equation corresponding to the nonlinear evolution relationship, an observation prediction equation and an observation Jacobian matrix are established;

[0138] According to the state covariance prediction equation and the observation Jacobian matrix, the Kalman gain value of the nonlinear state transition equation is calculated;

[0139] Based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, the state prediction equation is updated and solved, and the patient's movement intention is further predicted.

[0140] The expression of the state prediction equation corresponding to the nonlinear evolution relationship is:

[0141]

[0142] In the formula, represents the state vector at time t+1 predicted based on the information at time t, including eye movement position, muscle strength, torque and other dynamic parameters, represents a nonlinear state transition function, represents t the optimal state estimation value of the state after filtering and correction at time t;

[0143] The expression of the state transition Jacobian matrix is:

[0144]

[0145] In the formula, represents the state transition Jacobian matrix, representing the linearization gradient of the state transition function at the current estimation point, and is used to approximate the nonlinear function to a linear transformation;

[0146] The expression of the state covariance prediction equation is:

[0147]

[0148] In the formula, represents the covariance at time t+1 predicted based on the information at time t, denotes the optimal state covariance of the state after filtering correction at time t, denotes the compensation value of the rehabilitation robot,

[0149] The expression of the observation prediction equation is:

[0150]

[0151] In the formula, denotes the observation value at time t+1 predicted based on the information at time t, which corresponds to the theoretical output of the muscle strength sensor and the torque sensor, and the observation prediction equation is used to observe whether the motion trajectory of the rehabilitation robot meets the theoretical requirements, denotes the observation function, which is used to map the state vector to the sensor measurement space, and needs to consider the physical characteristics and coupling relationship of the sensor;

[0152] The expression of the observation Jacobian matrix is:

[0153]

[0154] In the formula, denotes the observation Jacobian matrix, which represents the linearized gradient of the observation function at the predicted state point, and is used to calculate the sensitivity of the state to the observation;

[0155] The expression of the Kalman gain value is:

[0156]

[0157] In the formula, denotes the Kalman gain value of the nonlinear state transition equation at time t+1, denotes the observation noise covariance matrix, which is used to describe the measurement error of the sensor, such as electromagnetic interference of the muscle strength signal and zero drift of the torque sensor;

[0158] The expression of the state prediction equation update is:

[0159]

[0160] In the formula, denotes the optimal state estimation value of the state after filtering correction at time t+1, and denotes the motion intention position of the patient, denotes the optimal observation value of the state after correction at time t.

[0161] S50, control the motion trajectory of the rehabilitation robot according to the predicted motion intention of the patient.

[0162] Further, the upper limb rehabilitation robot control method based on multi-modal data further comprises: obtaining the updated covariance of the state prediction equation, and the expression of the updated covariance of the state prediction equation is:

[0163]

[0164] wherein, denotes a unit matrix.

[0165] In summary, the upper limb rehabilitation robot control method based on multi-modal data provided in the application first arranges torque sensors at each joint of the robot, arranges muscle force sensors at the end of the patient's upper limb, and arranges a head-mounted sensor in the head-mounted device of the patient; acquires a torque signal set, an end-of-upper-limb muscle force action signal set, and an eye movement recognition signal set through the torque sensors, muscle force sensors, and head-mounted sensors, respectively; builds a patient motion intention prediction model through the collected multi-modal data, predicts the patient's motion intention at the next moment, and controls the motion trajectory of the upper limb rehabilitation robot according to the patient's motion intention, effectively improving the effect of the rehabilitation robot assisting the patient in rehabilitation, and being suitable for wide promotion.

[0166] Embodiment Two

[0167] The embodiment provides an upper limb rehabilitation robot control system based on multi-modal data, comprising:

[0168] A first acquisition module is configured to acquire human-machine interaction forces corresponding to each joint based on torque sensors arranged at joints of the upper limb rehabilitation robot, acquire torque signals corresponding to the human-machine interaction forces of each joint according to a torque Jacobian matrix, and obtain a torque signal set.

[0169] A second acquisition module is configured to acquire implicit neurons of the end of the patient's upper limb, acquire original muscle force data based on muscle force sensors arranged at the end of the patient's upper limb, and acquire an end-of-upper-limb muscle force action signal set according to the implicit neurons and the original muscle force data.

[0170] A third acquisition module is configured to acquire eye movement data information of the patient according to a head-mounted sensor of the patient, perform position compensation on the eye movement data information to eliminate boundary points in the eye movement data information, and obtain an eye movement recognition signal set.

[0171] A solving module is configured to build a multi-modal state vector according to the torque signal set, the end-of-upper-limb muscle force action signal set, and the eye movement recognition signal set, build a nonlinear state transition equation according to the multi-modal state vector and solve the nonlinear state transition equation, and predict the motion intention of the patient according to a solving result.

[0172] A control module is configured to control a motion trajectory of the rehabilitation robot according to the predicted motion intention of the patient.

[0173] Preferably, the torque signal set has an expression as follows:

[0174]

[0175] wherein, denotes the torque signal set at time t, denotes the joint the torque signal at time t, denotes the joint number;

[0176]

[0177] wherein, denotes the torque Jacobian matrix, denotes the joint the human-robot interaction force collected by the torque sensor at time t.

[0178] Preferably, the expression of the muscle force action signal set is:

[0179]

[0180] wherein, denotes the muscle force action signal set at time t, denotes the number of muscle force sensors arranged at the end of the upper limb, denotes the number of hidden neurons, denotes the weight vector between the hidden layer neurons and the output neurons, denotes the original muscle force data;

[0181]

[0182] wherein, denotes the muscle force sensor the original muscle force data collected at time t.

[0183] Preferably, the step of collecting the eye movement data information of the patient according to the sensor worn on the head of the patient, positionally compensating the eye movement data information to eliminate the boundary points in the eye movement data information, and obtaining an eye movement recognition signal set is:

[0184] According to the eye movement data information of the patient collected by the sensor worn on the head of the patient, the neighborhood of the fixation point in the eye movement data information is set, and the core object in the fixation point is determined;

[0185] Based on the determined core object, the density attribute of the fixation point is judged to eliminate the boundary points in the eye movement data information;

[0186] According to the density attribute relationship of the fixation point, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

[0187] Preferably, the expression of the neighborhood of the fixation point is:

[0188]

[0189] wherein, denotes a specified domain, denotes a gaze point at time t, and a distance between the gaze point and the gaze point denotes a distance threshold;

[0190] if a neighborhood of the gaze point satisfies:

[0191]

[0192] then the gaze point is a core object, wherein, denotes a neighborhood gaze point number threshold;

[0193] an expression of the eye movement recognition signal set is:

[0194]

[0195] wherein, denotes an eye movement recognition signal set at time t, denotes a gaze point corresponding eye movement recognition signal, denotes a gaze point number;

[0196]

[0197] wherein, denotes an initial eye movement recognition signal, denotes a gaze point corresponding eye movement data coefficient.

[0198] Preferably, an expression of the multi-modal state vector is:

[0199]

[0200] wherein, denotes a multi-modal state vector of the rehabilitation robot at time t, respectively denote a torque signal set, a muscle force action signal set and an eye movement recognition signal set at time t, respectively denote a derivative of the torque signal set, the muscle force action signal set and the eye movement recognition signal set with respect to time at time t;

[0201] an expression of the nonlinear state transition equation is:

[0202] ​

[0203] In the formula, represents a multi-modal state vector of the rehabilitation robot at t+1 time, represents a nonlinear state transition function, represents a preset instruction control vector of external input, represents a control process noise.

[0204] Preferably, the step of establishing a nonlinear state transition equation according to the multi-modal state vector and solving, and predicting the motion intention of the patient according to the solving result is:

[0205] Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from t time to t+1 time is established;

[0206] According to the state prediction equation corresponding to the nonlinear evolution relationship, a state transition Jacobian matrix is obtained, and a state covariance prediction equation is established according to the state transition Jacobian matrix;

[0207] According to the state prediction equation corresponding to the nonlinear evolution relationship, an observation prediction equation and an observation Jacobian matrix are established;

[0208] According to the state covariance prediction equation and the observation Jacobian matrix, the Kalman gain value of the nonlinear state transition equation is calculated;

[0209] Based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from t time to t+1 time and the observation prediction equation, the state prediction equation is updated and solved, and then the motion intention of the patient is predicted.

[0210] Preferably, the expression of the state prediction equation corresponding to the nonlinear evolution relationship is:

[0211]

[0212] In the formula, represents a state vector at t+1 time predicted based on t time information, represents a nonlinear state transition function, represents t the optimal state estimation value of the state after filtering correction at t time;

[0213] The expression of the state transition Jacobian matrix is:

[0214]

[0215] In the formula, represents a state transition Jacobian matrix;

[0216] The expression of the state covariance prediction equation is:

[0217]

[0218] In the formula, represents the t+1 time covariance predicted based on the t time information, represents the t time filtered and corrected state optimal state covariance, represents the compensation value of the rehabilitation robot;

[0219] The expression of the observation prediction equation is:

[0220]

[0221] In the formula, represents the t+1 time observation value predicted based on the t time information, represents the observation function;

[0222] The expression of the observation Jacobian matrix is:

[0223]

[0224] In the formula, represents the observation Jacobian matrix, representing the linearized gradient of the observation function at the predicted state point, and being used to calculate the sensitivity of the state to the observation;

[0225] The expression of the Kalman gain value is:

[0226]

[0227] In the formula, represents the t+1 time nonlinear state transition equation Kalman gain value, represents the observation noise covariance matrix, used to describe the sensor measurement error;

[0228] The expression of the state prediction equation update is:

[0229]

[0230] In the formula, represents the t+1 time filtered and corrected state optimal state estimation value, representing the motion intention position of the patient, represents the t time corrected state optimal observation value.

[0231] Preferably, the upper limb rehabilitation robot control system based on multi-modal data further comprises:

[0232] The acquisition module is configured to acquire the updated covariance of the state prediction equation, and the expression of the updated covariance of the state prediction equation is:

[0233]

[0234] In the formula, represents a unit matrix.

[0235] Embodiment three

[0236] Embodiment three of the present application provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the upper limb rehabilitation robot control method based on multi-modal data as described above.

[0237] Embodiment four

[0238] The present application also provides a computer, please refer to Figure 3 , which is a computer in embodiment four of the present application, comprising a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20, wherein the processor 20 implements the upper limb rehabilitation robot control method based on multi-modal data as described above when executing the computer program 30.

[0239] The memory 10 includes at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 10 can be an internal storage unit of the computer, such as the hard disk of the computer. In other embodiments, the memory 10 can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 10 can include both an internal storage unit of the computer and an external storage device. The memory 10 can be used not only to store application software and various data installed on the computer, but also to temporarily store data that has been output or will be output.

[0240] The processor 20 can be an electronic control unit (ECU), a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip, which is used to run program codes or process data stored in the memory 10, such as to execute access restriction programs, in some embodiments.

[0241] It should be noted that Figure 3 The structure shown does not constitute a limitation on the computer, and in other embodiments, the computer can include fewer or more components than shown, or combine certain components, or different component arrangements.

[0242] Those skilled in the art will appreciate that the logic and / or steps represented in the flow diagrams, or otherwise described herein, for example, can be thought of as a list of executable instructions for implementing logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or a combination of both. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0243] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, and stored in a computer memory in a form that is then employable by a computer.

[0244] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: a discrete logic circuit having logic gates for implementing logic functions upon data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), or the like.

[0245] The technical features of the above-described embodiments can be combined in any manner, and in order to make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, however, as long as the combinations of the technical features do not contradict each other, it should be considered that they are within the scope of the present specification.

[0246] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. An upper limb rehabilitation robot control system based on multimodal data, characterized in that: include: The first acquisition module is configured to acquire the human-machine interaction force corresponding to each joint based on torque sensors arranged at several joints of the upper limb rehabilitation robot, obtain the torque signal corresponding to the human-machine interaction force of each joint based on the torque Jacobian matrix, and obtain a torque signal set; The second acquisition module is used to obtain hidden neurons at the end of the patient's upper limb, collect raw muscle force data based on muscle force sensors arranged at the end of the patient's upper limb, and obtain a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; a third acquisition module, configured to collect eye movement data information of the patient using a sensor worn on the patient's head, perform position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtain an eye movement recognition signal set; a solution module, configured to establish a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establish and solve a nonlinear state transfer equation based on the multimodal state vector, and predict the patient's movement intention based on the solution result; For the solver module: The expression of the multimodal state vector is: Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. They represent the time derivatives of the torque signal set, muscle action signal set, and eye movement recognition signal set at time t respectively; The expression of the nonlinear state transfer equation is: Where, represents the multimodal state vector of the rehabilitation robot at time t+1, represents the nonlinear state transfer function, Represents the preset instruction control vector of external input, represents the control process noise; The steps of establishing and solving a nonlinear state transfer equation based on the multimodal state vector and predicting the patient's movement intention based on the solution result are as follows: Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established; The state transfer Jacobian matrix is ​​obtained according to the state prediction equation corresponding to the nonlinear evolution relationship, and the state covariance prediction equation is established according to the state transfer Jacobian matrix; According to the state prediction equation corresponding to the nonlinear evolution relationship, the observation prediction equation and the observation Jacobian matrix are established; Calculate the Kalman gain value of the nonlinear state transfer equation according to the state covariance prediction equation and the observation Jacobian matrix; Based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, the state prediction equation is updated and solved to predict the patient's movement intention; The expression of the state prediction equation corresponding to the nonlinear evolution relationship is: Where, represents the state vector at time t+1 predicted based on the information at time t, represents the nonlinear state transfer function, express t The optimal state estimate after filtering and correction at each moment; The expression of the state transfer Jacobian matrix is: Where, represents the state transition Jacobian matrix; The expression of the state covariance prediction equation is: Where, represents the covariance at time t+1 predicted based on the information at time t, represents the optimal state covariance after filtering and correction at time t, Indicates the compensation value of the rehabilitation robot; The expression of the observation prediction equation is: Where, represents the observation value at time t+1 predicted based on the information at time t, represents the observation function; The expression of the observation Jacobian matrix is: Where, Represents the observation Jacobian matrix, which characterizes the linearized gradient of the observation function at the predicted state point and is used to calculate the sensitivity of the state to the observation; The expression of the Kalman gain value is: Where, Represents the Kalman gain value of the nonlinear state transfer equation at time t+1, represents the observation noise covariance matrix, which is used to describe the sensor measurement error; The expression for updating the state prediction equation is: Where, represents the optimal state estimate after filtering and correction at t+1, represents the patient's movement intention position, It represents the optimal observation value of the state after correction at time t; The control module is used to control the motion trajectory of the rehabilitation robot according to the predicted patient's motion intention.

2. The upper limb rehabilitation robot control system based on multimodal data according to claim 1, characterized in that: For the first acquisition module: The expression of the torque signal set is: Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints; Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

3. The upper limb rehabilitation robot control system based on multimodal data according to claim 2, characterized in that: For the second acquisition module: The expression of the muscle action signal set is: Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, Represents raw muscle strength data; Where, Indicates muscle force sensor Raw muscle force data collected at time t.

4. The upper limb rehabilitation robot control system based on multimodal data according to claim 3, characterized in that: In the third acquisition module, the steps of collecting the patient's eye movement data information based on the sensor worn on the patient's head, performing position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtaining an eye movement recognition signal set are as follows: Collecting the patient's eye movement data using a sensor worn on the patient's head, setting the neighborhood of the gaze point in the eye movement data, and determining the core object in the gaze point; Determine the gaze point density attribute based on the determined core object to eliminate boundary points in the eye movement data information; According to the density attribute relationship of the gaze points, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

5. The upper limb rehabilitation robot control system based on multimodal data according to claim 4, characterized in that: The expression of the neighborhood of the gaze point is: Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold; If the gaze point Neighborhood satisfy: The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations; The expression of the eye movement recognition signal set is: Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations; Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

6. The upper limb rehabilitation robot control system based on multimodal data according to claim 5, characterized in that: The solution module also includes: Obtain the updated covariance of the state prediction equation. The expression of the updated covariance of the state prediction equation is: Where, Represents the identity matrix.

Images