Upper limb rehabilitation robot control method and system based on multi-modal data

Abstract

The invention provides an upper limb rehabilitation robot control method and system based on multi-modal data. Torque signals corresponding to all joints of the rehabilitation robot are collected, and a torque signal set is obtained; acquiring implicit neurons and original muscle force data of the tail end of the upper limb of the patient, and acquiring a muscle force action signal set of the tail end of the upper limb of the patient according to the implicit neurons and the original muscle force data; collecting eye movement data information of a patient, and performing position compensation on the eye movement data information to obtain an eye movement recognition signal set; establishing a multi-modal state vector according to the torque signal set, the tail end muscle force action signal set and the eye movement recognition signal set, establishing a nonlinear state transition equation according to the multi-modal state vector and solving, and predicting the motion intention of the patient according to a solving result; the motion track of the rehabilitation robot is controlled according to the predicted motion intention of the patient, and the control method is good in rehabilitation effect and suitable for large-scale popularization.

Description

Technical Field

[0001] The present invention relates to the technical field of medical devices, and in particular to a method and system for controlling an upper limb rehabilitation robot based on multimodal data. Background Art

[0002] Upper limb rehabilitation training mainly focuses on training the shoulder, elbow, wrist, and metacarpophalangeal and finger functions. Depending on the severity of the patient's illness, upper limb rehabilitation training includes active training and passive assisted training under the guidance of a rehabilitation trainer.

[0003] Normally, mild patients adopt active training, and patients carry out corresponding rehabilitation training according to their subjective consciousness, while severe patients adopt passive assisted training. Currently, common passive assisted training mainly includes nurse interactive training and robot-assisted training. Nurse interactive training requires high experience of nurses and is difficult to promote; robot-assisted training The robot's movement trajectory is set in advance, and there is a lack of interaction between the robot and the patient, which makes it difficult to meet the patient's actual needs; how to predict the patient's movement intention at the next moment based on the patient's current movement situation, and control the movement trajectory of the rehabilitation robot according to the patient's movement intention to improve the patient's rehabilitation effect has gradually become a hot issue in assisted rehabilitation. Summary of the Invention

[0004] Based on this, the purpose of the present invention is to provide an upper limb rehabilitation robot control method and system based on multimodal data to solve the technical problems existing in the prior art.

[0005] The present invention proposes a control method for an upper limb rehabilitation robot based on multimodal data, comprising: The torque sensors arranged at several joints of the upper limb rehabilitation robot collect the human-machine interaction force corresponding to each joint, and obtain the torque signal corresponding to the human-machine interaction force of each joint according to the torque Jacobian matrix to obtain a torque signal set; Obtaining hidden neurons at the end of the patient's upper limb, collecting raw muscle force data based on a muscle force sensor arranged at the end of the patient's upper limb, and obtaining a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; collecting eye movement data of the patient using a sensor worn on the patient's head, performing position compensation on the eye movement data to remove boundary points in the eye movement data, and obtaining an eye movement recognition signal set; establishing a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establishing and solving a nonlinear state transfer equation based on the multimodal state vector, and predicting the patient's movement intention based on the solution result; The motion trajectory of the rehabilitation robot is controlled according to the predicted patient's motion intention.

[0006] Preferably, the torque signal set is expressed as:

[0007] Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints;

[0008] Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

[0009] Preferably, the expression of the muscle action signal set is:

[0010] Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, Represents raw muscle strength data;

[0011] Where, Indicates muscle force sensor Raw muscle force data collected at time t.

[0012] Preferably, the step of collecting the patient's eye movement data information according to the sensor worn on the patient's head, performing position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtaining the eye movement recognition signal set comprises: Collecting the patient's eye movement data using a sensor worn on the patient's head, setting the neighborhood of the gaze point in the eye movement data, and determining the core object in the gaze point; Determine the gaze point density attribute based on the determined core object to eliminate boundary points in the eye movement data information; According to the density attribute relationship of the gaze points, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

[0013] Preferably, the neighborhood of the gaze point is expressed as:

[0014] Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold; If the gaze point Neighborhood satisfy:

[0015] The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations; The expression of the eye movement recognition signal set is:

[0016] Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations;

[0017] Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

[0018] Preferably, the expression of the multimodal state vector is:

[0019] Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. They represent the time derivatives of the torque signal set, muscle action signal set, and eye movement recognition signal set at time t respectively; The expression of the nonlinear state transfer equation is:

[0020] Where, represents the multimodal state vector of the rehabilitation robot at time t+1, represents the nonlinear state transfer function, Represents the preset instruction control vector of external input, Represents the control process noise.

[0021] Preferably, the steps of establishing and solving a nonlinear state transfer equation based on the multimodal state vector and predicting the patient's movement intention based on the solution result are: Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established; The state transfer Jacobian matrix is ​​obtained according to the state prediction equation corresponding to the nonlinear evolution relationship, and the state covariance prediction equation is established according to the state transfer Jacobian matrix; According to the state prediction equation corresponding to the nonlinear evolution relationship, the observation prediction equation and the observation Jacobian matrix are established; Calculate the Kalman gain value of the nonlinear state transfer equation according to the state covariance prediction equation and the observation Jacobian matrix; The state prediction equation is updated and solved based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, thereby predicting the patient's movement intention.

[0022] Preferably, the state prediction equation corresponding to the nonlinear evolution relationship is expressed as:

[0023] Where, represents the state vector at time t+1 predicted based on the information at time t, represents the nonlinear state transfer function, express t The optimal state estimate after filtering and correction at each moment; The expression of the state transfer Jacobian matrix is:

[0024] Where, represents the state transition Jacobian matrix; The expression of the state covariance prediction equation is:

[0025] Where, represents the covariance at time t+1 predicted based on the information at time t, represents the optimal state covariance after filtering and correction at time t, Represents the compensation value of the rehabilitation robot; The expression of the observation prediction equation is:

[0026] Where, represents the observation value at time t+1 predicted based on the information at time t, represents the observation function; The expression of the observation Jacobian matrix is:

[0027] Where, Represents the observation Jacobian matrix, which characterizes the linearized gradient of the observation function at the predicted state point and is used to calculate the sensitivity of the state to the observation; The expression of the Kalman gain value is:

[0028] Where, Represents the Kalman gain value of the nonlinear state transfer equation at time t+1, represents the observation noise covariance matrix, which is used to describe the sensor measurement error; The expression for updating the state prediction equation is:

[0029] Where, represents the optimal state estimate after filtering and correction at t+1, represents the patient's movement intention position, It represents the optimal observation value of the state after correction at time t.

[0030] Preferably, the upper limb rehabilitation robot control method based on multimodal data further includes: obtaining the updated covariance of the state prediction equation, and the expression of the updated covariance of the state prediction equation is:

[0031] Where, Represents the identity matrix.

[0032] The present invention also proposes an upper limb rehabilitation robot control system based on multimodal data, comprising: The first acquisition module is configured to acquire the human-machine interaction force corresponding to each joint based on torque sensors arranged at several joints of the upper limb rehabilitation robot, obtain the torque signal corresponding to the human-machine interaction force of each joint based on the torque Jacobian matrix, and obtain a torque signal set; The second acquisition module is used to obtain hidden neurons at the end of the patient's upper limb, collect raw muscle force data based on muscle force sensors arranged at the end of the patient's upper limb, and obtain a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; a third acquisition module, configured to collect eye movement data information of the patient using a sensor worn on the patient's head, perform position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtain an eye movement recognition signal set; a solution module, configured to establish a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establish and solve a nonlinear state transfer equation based on the multimodal state vector, and predict the patient's movement intention based on the solution result; The control module is used to control the motion trajectory of the rehabilitation robot according to the predicted patient's motion intention.

[0033] The beneficial effects of the present invention compared with the prior art are as follows: the upper limb rehabilitation robot control method based on multimodal data proposed in this application first arranges torque sensors at each joint of the robot, arranges muscle force sensors at the end of the patient's upper limbs, and arranges head-worn sensors in the patient's head-worn device; the torque signal set, the patient's upper limb end muscle force action signal set, and the eye movement recognition signal set are respectively obtained through the torque sensor, muscle force sensor, and head-worn sensor; a prediction model of the patient's movement intention is established through the collected multimodal data, the patient's movement intention at the next moment is predicted, and the movement trajectory of the upper limb rehabilitation robot is controlled according to the patient's movement intention, which effectively improves the effect of the rehabilitation robot in assisting the patient in rehabilitation, and is suitable for large-scale promotion.

[0034] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a flowchart of the upper limb rehabilitation robot control method based on multimodal data in Example 1 of the present invention; Figure 2 This is a schematic diagram of signal acquisition in the first embodiment of the present invention; Figure 3 This is a structural block diagram of a computer in Embodiment 4 of the present invention.

[0036] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0037] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the accompanying drawings. The drawings illustrate several embodiments of the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the present invention.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The terms used in this specification of the present invention are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0039] Example 1 See also Figure 1 , which shows an upper limb rehabilitation robot control method based on multimodal data in embodiment 1 of the present invention. Specifically, the upper limb rehabilitation robot control method based on multimodal data includes steps S10 to S50: S10, collecting human-machine interaction forces corresponding to the joints based on torque sensors arranged at several joints of the upper limb rehabilitation robot, obtaining torque signals corresponding to the human-machine interaction forces of the joints based on the torque Jacobian matrix, and obtaining a torque signal set; In specific implementation, as the patient's upper limb motor function gradually recovers, the patient needs to actively participate in the training process during the upper limb rehabilitation training. Therefore, human-computer interaction training should be carried out according to the patient's movement intention, which requires the control process of the upper limb rehabilitation robot to judge the patient's movement intention in advance; Figure 2 As shown, patients need to collect multimodal signals during the upper limb rehabilitation process. The torque sensor is arranged at the joints of the upper limb rehabilitation robot to obtain torque signal sets at different times; the muscle force sensor is arranged at the end of the patient's upper limb to obtain the patient's muscle force signal, and the head-worn sensor is arranged in the device worn by the patient to obtain scanning data when the patient turns his head.

[0040] The expression of the torque signal set is:

[0041] Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints;

[0042] Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

[0043] S20, obtaining hidden neurons at the end of the patient's upper limb, collecting raw muscle force data based on a muscle force sensor arranged at the end of the patient's upper limb, and obtaining a muscle force action signal set at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; Optionally, based on the contact status between the upper limb rehabilitation robot and the patient's upper limb extremities, the end muscle force signal can be collected to quickly and efficiently identify the patient's movement intention.

[0044] Optionally, the expression of the muscle action signal set is:

[0045] Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, Represents raw muscle strength data;

[0046] Where, Indicates muscle force sensor Raw muscle force data collected at time t.

[0047] S30, collecting eye movement data information of the patient according to the sensor worn on the patient's head, performing position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtaining an eye movement recognition signal set; Optionally, the step of collecting the patient's eye movement data information based on the sensor worn on the patient's head, performing position compensation on the eye movement data information to eliminate boundary points in the eye movement data information, and obtaining the eye movement recognition signal set is: Collecting the patient's eye movement data using a sensor worn on the patient's head, setting the neighborhood of the gaze point in the eye movement data, and determining the core object in the gaze point; Determine the gaze point density attribute based on the determined core object to eliminate boundary points in the eye movement data information; According to the density attribute relationship of the fixation points, the boundary points in the eye movement data are eliminated, and the density-connected sample set is derived as the eye movement recognition signal set; In principle, the specific steps for determining the gaze point density attribute are: If gaze point A is in the neighborhood of gaze point B, and gaze point B is a core object, then A is said to be directly reached by B’s density, that is, A is directly “covered” by the dense area where B is located; If there is a string of fixations B→C→D→…→A, where each fixation point can directly reach the next point, such as B directly reaching C, C directly reaching D, and so on, and the last point directly reaching A, then A is said to be density-reachable by B; that is, A indirectly belongs to the dense area where B is located through a series of core objects; If there exists a core object O such that both gaze point A and gaze point B are density-reachable from O, then A and B are said to be density-connected; that is, A and B belong to the same dense region; Based on the density attribute relationship of the fixation points, the boundary points in the eye movement data information are eliminated, and the density-connected sample set is derived as the eye movement recognition signal set; the effective eye movement signals are extracted, and through the above density relationship analysis, points that are neither core objects nor can be associated with any core objects through density reachability may be eye movement errors or interferences, and are considered to be boundary points and will be eliminated. Eliminating the boundary points in the eye movement data information is conducive to solving problems such as displacement drift caused by eye scanning during the signal acquisition process; the remaining fixation point set after eliminating the boundary points is the eye movement recognition signal set.

[0048] The expression of the neighborhood of the gaze point is:

[0049] Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold; If the gaze point Neighborhood satisfy:

[0050] The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations; The expression of the eye movement recognition signal set is:

[0051] Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations;

[0052] Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

[0053] S40, establishing a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establishing and solving a nonlinear state transfer equation based on the multimodal state vector, and predicting the patient's movement intention based on the solution result; Optionally, in order to better adapt to the movements of patients with upper limb hemiplegia and improve the human-computer interaction performance of the upper limb rehabilitation robot, based on the joint torque signal set, muscle action signal set, and eye movement recognition signal set collected by the above sensors and corresponding processing of the collected information, an efficient movement intention recognition algorithm can be designed based on the corresponding features to determine the patient's movement intention. In this embodiment, the joint torque signal, muscle action signal, and eye movement recognition signal are respectively:

[0054]

[0055]

[0056] Based on the signal data collected above, multimodal data information of the torque sensor, muscle force sensor, and head-worn sensor can be obtained after corresponding processing. Based on the data information collected above, their multimodal state vectors are established; the expression of the multimodal state vector is:

[0057] Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. They represent the time derivatives of the torque signal set, muscle action signal set, and eye movement recognition signal set at time t respectively; Taking into account the motion trajectory characteristics of the upper limb rehabilitation robot, a nonlinear state transfer equation is established based on the extended Kalman filter (EKF); the expression of the nonlinear state transfer equation is:

[0058] Where, Represents the multimodal state vector of the rehabilitation robot at time t+1, which is used to predict the patient's movement intention at the next moment. represents the nonlinear state transfer function, Represents the preset instruction control vector of external input, represents the control process noise.

[0059] Optionally, the steps of establishing and solving a nonlinear state transfer equation based on the multimodal state vector, and predicting the patient's movement intention based on the solution result are: Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established; The state transfer Jacobian matrix is ​​obtained according to the state prediction equation corresponding to the nonlinear evolution relationship, and the state covariance prediction equation is established according to the state transfer Jacobian matrix; According to the state prediction equation corresponding to the nonlinear evolution relationship, the observation prediction equation and the observation Jacobian matrix are established; Calculate the Kalman gain value of the nonlinear state transfer equation according to the state covariance prediction equation and the observation Jacobian matrix; The state prediction equation is updated and solved based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, thereby predicting the patient's movement intention.

[0060] The expression of the state prediction equation corresponding to the nonlinear evolution relationship is:

[0061] Where, Represents the state vector at time t+1 predicted based on the information at time t, including dynamic parameters such as eye position, muscle strength, torque, etc. represents the nonlinear state transfer function, express t The optimal state estimate after filtering and correction at each moment; The expression of the state transfer Jacobian matrix is:

[0062] Where, Represents the state transfer Jacobian matrix, which characterizes the linearized gradient of the state transfer function at the current estimation point and is used to approximate nonlinear functions into linear transformations; The expression of the state covariance prediction equation is:

[0063] Where, represents the covariance at time t+1 predicted based on the information at time t, represents the optimal state covariance after filtering and correction at time t, Indicates the compensation value of the rehabilitation robot; The expression of the observation prediction equation is:

[0064] Where, It represents the observation value at time t+1 predicted based on the information at time t, corresponding to the theoretical output of the muscle force sensor and torque sensor. The observation prediction equation is used to observe whether the motion trajectory of the rehabilitation robot meets the theoretical requirements. Represents the observation function, which is used to map the state vector to the sensor measurement space, taking into account the physical characteristics and coupling relationship of the sensor; The expression of the observation Jacobian matrix is:

[0065] Where, Represents the observation Jacobian matrix, which characterizes the linearized gradient of the observation function at the predicted state point and is used to calculate the sensitivity of the state to the observation; The expression of the Kalman gain value is:

[0066] Where, Represents the Kalman gain value of the nonlinear state transfer equation at time t+1, Represents the observation noise covariance matrix, which is used to describe sensor measurement errors, such as electromagnetic interference of muscle force signals and zero drift of torque sensors; The expression for updating the state prediction equation is:

[0067] Where, represents the optimal state estimate after filtering and correction at t+1, represents the patient's movement intention position, It represents the optimal observation value of the state after correction at time t.

[0068] S50 , controlling the motion trajectory of the rehabilitation robot according to the predicted patient's motion intention.

[0069] Furthermore, the upper limb rehabilitation robot control method based on multimodal data further includes: obtaining the updated covariance of the state prediction equation, and the expression of the updated covariance of the state prediction equation is:

[0070] Where, Represents the identity matrix.

[0071] In summary, the upper limb rehabilitation robot control method based on multimodal data proposed in this application first arranges torque sensors at each joint of the robot, muscle force sensors at the end of the patient's upper limbs, and head-worn sensors in the patient's head-worn device; the torque signal set, the patient's upper limb end muscle force action signal set, and the eye movement recognition signal set are respectively obtained through the torque sensor, muscle force sensor, and head-worn sensor; a prediction model of the patient's movement intention is established through the collected multimodal data, the patient's movement intention at the next moment is predicted, and the movement trajectory of the upper limb rehabilitation robot is controlled according to the patient's movement intention, which effectively improves the effect of the rehabilitation robot in assisting patients in rehabilitation and is suitable for large-scale promotion.

[0072] Example 2 This embodiment provides an upper limb rehabilitation robot control system based on multimodal data, including: The first acquisition module is configured to acquire the human-machine interaction force corresponding to each joint based on torque sensors arranged at several joints of the upper limb rehabilitation robot, obtain the torque signal corresponding to the human-machine interaction force of each joint based on the torque Jacobian matrix, and obtain a torque signal set; The second acquisition module is used to obtain hidden neurons at the end of the patient's upper limb, collect raw muscle force data based on muscle force sensors arranged at the end of the patient's upper limb, and obtain a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; a third acquisition module, configured to collect eye movement data information of the patient using a sensor worn on the patient's head, perform position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtain an eye movement recognition signal set; a solution module, configured to establish a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establish and solve a nonlinear state transfer equation based on the multimodal state vector, and predict the patient's movement intention based on the solution result; The control module is used to control the motion trajectory of the rehabilitation robot according to the predicted patient's motion intention.

[0073] Preferably, the expression of the torque signal set is:

[0074] Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints;

[0075] Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

[0076] Preferably, the expression of the muscle action signal set is:

[0077] Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, Represents raw muscle strength data;

[0078] Where, Indicates muscle force sensor Raw muscle force data collected at time t.

[0079] Preferably, the step of collecting the patient's eye movement data information according to the sensor worn on the patient's head, performing position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtaining the eye movement recognition signal set comprises: Collecting the patient's eye movement data using a sensor worn on the patient's head, setting the neighborhood of the gaze point in the eye movement data, and determining the core object in the gaze point; Determine the gaze point density attribute based on the determined core object to eliminate boundary points in the eye movement data information; According to the density attribute relationship of the gaze points, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

[0080] Preferably, the neighborhood of the gaze point is expressed as:

[0081] Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold; If the gaze point Neighborhood satisfy:

[0082] The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations; The expression of the eye movement recognition signal set is:

[0083] Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations;

[0084] Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

[0085] Preferably, the expression of the multimodal state vector is:

[0086] Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. They represent the time derivatives of the torque signal set, muscle action signal set, and eye movement recognition signal set at time t respectively; The expression of the nonlinear state transfer equation is:

[0087] Where, represents the multimodal state vector of the rehabilitation robot at time t+1, represents the nonlinear state transfer function, Represents the preset instruction control vector of external input, Represents the control process noise.

[0088] Preferably, the steps of establishing and solving a nonlinear state transfer equation based on the multimodal state vector and predicting the patient's movement intention based on the solution result are: Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established; The state transfer Jacobian matrix is ​​obtained according to the state prediction equation corresponding to the nonlinear evolution relationship, and the state covariance prediction equation is established according to the state transfer Jacobian matrix; According to the state prediction equation corresponding to the nonlinear evolution relationship, the observation prediction equation and the observation Jacobian matrix are established; Calculate the Kalman gain value of the nonlinear state transfer equation according to the state covariance prediction equation and the observation Jacobian matrix; The state prediction equation is updated and solved based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, thereby predicting the patient's movement intention.

[0089] Preferably, the state prediction equation corresponding to the nonlinear evolution relationship is expressed as:

[0090] Where, represents the state vector at time t+1 predicted based on the information at time t, represents the nonlinear state transfer function, express t The optimal state estimate after filtering and correction at each moment; The expression of the state transfer Jacobian matrix is:

[0091] Where, represents the state transition Jacobian matrix; The expression of the state covariance prediction equation is:

[0092] Where, represents the covariance at time t+1 predicted based on the information at time t, represents the optimal state covariance after filtering and correction at time t, Indicates the compensation value of the rehabilitation robot; The expression of the observation prediction equation is:

[0093] Where, represents the observation value at time t+1 predicted based on the information at time t, represents the observation function; The expression of the observation Jacobian matrix is:

[0094] Where, Represents the observation Jacobian matrix, which characterizes the linearized gradient of the observation function at the predicted state point and is used to calculate the sensitivity of the state to the observation; The expression of the Kalman gain value is:

[0095] Where, Represents the Kalman gain value of the nonlinear state transfer equation at time t+1, represents the observation noise covariance matrix, which is used to describe the sensor measurement error; The expression for updating the state prediction equation is:

[0096] Where, represents the optimal state estimate after filtering and correction at t+1, represents the patient's movement intention position, It represents the optimal observation value of the state after correction at time t.

[0097] Preferably, the upper limb rehabilitation robot control system based on multimodal data further includes: The acquisition module is used to obtain the covariance of the updated state prediction equation. The expression of the covariance of the updated state prediction equation is:

[0098] Where, Represents the identity matrix.

[0099] Example 3 A third embodiment of the present invention provides a storage medium having a computer program stored thereon. When the program is executed by a processor, the above-mentioned upper limb rehabilitation robot control method based on multimodal data is implemented.

[0100] Example 4 The present invention also provides a computer, see Figure 3 , shown is a computer in embodiment 4 of the present invention, including a memory 10, a processor 20, and a computer program 30 stored in the memory 10 and executable on the processor 20. When the processor 20 executes the computer program 30, the above-mentioned upper limb rehabilitation robot control method based on multimodal data is implemented.

[0101] The memory 10 includes at least one type of storage medium, including flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the memory 10 may be an internal storage unit of a computer, such as the computer's hard disk. In other embodiments, the memory 10 may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the memory 10 may include both an internal storage unit of the computer and an external storage device. The memory 10 can be used not only to store application software installed in the computer and various types of data, but also to temporarily store data that has been output or is about to be output.

[0102] Among them, in some embodiments, the processor 20 can be an electronic control unit (Electronic Control Unit, abbreviated as ECU, also known as a vehicle computer), a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chip, used to run the program code stored in the memory 10 or process data, such as executing access restriction programs.

[0103] It should be pointed out that Figure 3 The structure shown does not constitute a limitation of the computer. In other embodiments, the computer may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0104] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0105] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0106] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the aforementioned embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following technologies known in the art may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0107] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0108] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A control method for an upper limb rehabilitation robot based on multimodal data, characterized in that: include: The torque sensors arranged at several joints of the upper limb rehabilitation robot collect the human-machine interaction force corresponding to each joint, and obtain the torque signal corresponding to the human-machine interaction force of each joint according to the torque Jacobian matrix to obtain a torque signal set; Obtaining hidden neurons at the end of the patient's upper limb, collecting raw muscle force data based on a muscle force sensor arranged at the end of the patient's upper limb, and obtaining a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; collecting eye movement data of the patient using a sensor worn on the patient's head, performing position compensation on the eye movement data to remove boundary points in the eye movement data, and obtaining an eye movement recognition signal set; establishing a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establishing and solving a nonlinear state transfer equation based on the multimodal state vector, and predicting the patient's movement intention based on the solution result; The motion trajectory of the rehabilitation robot is controlled according to the predicted patient's motion intention.

2. The upper limb rehabilitation robot control method based on multimodal data according to claim 1, characterized in that: The expression of the torque signal set is: Where, represents the torque signal set at time t, Indicates joints The torque signal at time t is: Indicates the number of joints; Where, represents the torque Jacobian matrix, Indicates joints Human-machine interaction force collected by the torque sensor at time t.

3. The upper limb rehabilitation robot control method based on multimodal data according to claim 2, characterized in that: The expression of the muscle action signal set is: Where, represents the muscle action signal set at time t, Indicates the number of muscle force sensors arranged at the ends of the upper limbs, represents the number of hidden neurons, Represents the hidden layer neurons and the weight vector between the output neurons, Represents raw muscle strength data; Where, Indicates muscle force sensor Raw muscle force data collected at time t.

4. The upper limb rehabilitation robot control method based on multimodal data according to claim 3, characterized in that: The step of collecting the patient's eye movement data information according to the sensor worn on the patient's head, performing position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtaining an eye movement recognition signal set is as follows: Collecting the patient's eye movement data using a sensor worn on the patient's head, setting the neighborhood of the gaze point in the eye movement data, and determining the core object in the gaze point; Determine the gaze point density attribute based on the determined core object to eliminate boundary points in the eye movement data information; According to the density attribute relationship of the gaze points, the boundary points in the eye movement data information are eliminated, and the density connected sample set is derived as the eye movement recognition signal set.

5. The upper limb rehabilitation robot control method based on multimodal data according to claim 4, characterized in that: The expression of the neighborhood of the gaze point is: Where, Indicates the specified domain. represents the gaze point at time t and gaze point distance, represents the distance threshold; If the gaze point Neighborhood satisfy: The fixation point is the core object, among which, represents the threshold of the number of neighborhood fixations; The expression of the eye movement recognition signal set is: Where, represents the eye movement recognition signal set at time t, Indicates the gaze point The corresponding eye movement recognition signal, represents the number of fixations; Where, represents the initial eye movement recognition signal, Indicates the gaze point The corresponding eye movement data coefficients.

6. The upper limb rehabilitation robot control method based on multimodal data according to claim 5, characterized in that: The expression of the multimodal state vector is: Where, represents the multimodal state vector of the rehabilitation robot at time t, They represent the torque signal set, muscle action signal set, and eye movement recognition signal set at time t, respectively. They represent the time derivatives of the torque signal set, muscle action signal set, and eye movement recognition signal set at time t respectively; The expression of the nonlinear state transfer equation is: Where, represents the multimodal state vector of the rehabilitation robot at time t+1, represents the nonlinear state transfer function, Represents the preset instruction control vector of external input, Represents the control process noise.

7. The upper limb rehabilitation robot control method based on multimodal data according to claim 6, characterized in that: The steps of establishing and solving a nonlinear state transfer equation based on the multimodal state vector and predicting the patient's movement intention based on the solution result are as follows: Based on the dynamic characteristics of the physical system, a state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1 is established; The state transfer Jacobian matrix is ​​obtained according to the state prediction equation corresponding to the nonlinear evolution relationship, and the state covariance prediction equation is established according to the state transfer Jacobian matrix; According to the state prediction equation corresponding to the nonlinear evolution relationship, the observation prediction equation and the observation Jacobian matrix are established; Calculate the Kalman gain value of the nonlinear state transfer equation according to the state covariance prediction equation and the observation Jacobian matrix; The state prediction equation is updated and solved based on the Kalman gain value, the state prediction equation corresponding to the nonlinear evolution relationship from time t to time t+1, and the observation prediction equation, thereby predicting the patient's movement intention.

8. The upper limb rehabilitation robot control method based on multimodal data according to claim 7, characterized in that: The expression of the state prediction equation corresponding to the nonlinear evolution relationship is: Where, represents the state vector at time t+1 predicted based on the information at time t, represents the nonlinear state transfer function, express t The optimal state estimate after filtering and correction at each moment; The expression of the state transfer Jacobian matrix is: Where, represents the state transition Jacobian matrix; The expression of the state covariance prediction equation is: Where, represents the covariance at time t+1 predicted based on the information at time t, represents the optimal state covariance after filtering and correction at time t, Indicates the compensation value of the rehabilitation robot; The expression of the observation prediction equation is: Where, represents the observation value at time t+1 predicted based on the information at time t, represents the observation function; The expression of the observation Jacobian matrix is: Where, Represents the observation Jacobian matrix, which characterizes the linearized gradient of the observation function at the predicted state point and is used to calculate the sensitivity of the state to the observation; The expression of the Kalman gain value is: Where, Represents the Kalman gain value of the nonlinear state transfer equation at time t+1, represents the observation noise covariance matrix, which is used to describe the sensor measurement error; The expression for updating the state prediction equation is: Where, represents the optimal state estimate after filtering and correction at t+1, represents the patient's movement intention position, It represents the optimal observation value of the state after correction at time t.

9. The upper limb rehabilitation robot control method based on multimodal data according to claim 8, characterized in that: The upper limb rehabilitation robot control method based on multimodal data further includes: obtaining the updated covariance of the state prediction equation, wherein the expression of the updated covariance of the state prediction equation is: Where, Represents the identity matrix.

10. An upper limb rehabilitation robot control system based on multimodal data, characterized in that: include: The first acquisition module is configured to acquire the human-machine interaction force corresponding to each joint based on torque sensors arranged at several joints of the upper limb rehabilitation robot, obtain the torque signal corresponding to the human-machine interaction force of each joint based on the torque Jacobian matrix, and obtain a torque signal set; The second acquisition module is used to obtain hidden neurons at the end of the patient's upper limb, collect raw muscle force data based on muscle force sensors arranged at the end of the patient's upper limb, and obtain a set of muscle force action signals at the end of the patient's upper limb based on the hidden neurons and the raw muscle force data; a third acquisition module, configured to collect eye movement data information of the patient using a sensor worn on the patient's head, perform position compensation on the eye movement data information to remove boundary points in the eye movement data information, and obtain an eye movement recognition signal set; a solution module, configured to establish a multimodal state vector based on the torque signal set, the terminal muscle action signal set, and the eye movement recognition signal set, establish and solve a nonlinear state transfer equation based on the multimodal state vector, and predict the patient's movement intention based on the solution result; The control module is used to control the motion trajectory of the rehabilitation robot according to the predicted patient's motion intention.

Images