Digital twin upper limb rehabilitation training robot and method considering patient emotion

Abstract

The application discloses a digital twin upper limb rehabilitation training robot and method considering patient emotion, which comprises: a teleoperation master robot for obtaining accurate action information of a trainer; at least one wearable exoskeleton mechanical arm for obtaining force, heart rate, electromyographic signal and facial expression of the patient respectively, restoring the action of the trainer and making the same movement with the patient; a computer system carrying a digital twin, comprising a digital twin model established according to the connecting rod relationship of the mechanical arm, and a man-machine interaction module and a voice broadcast module; according to different training stages, the action of the trainer is learned and output to guide the exoskeleton mechanical arm to replicate the action. In the training process, the trainer can participate and help the patient to restore the action of the trainer through the teleoperation teaching mode, and at the same time, the patient emotion is considered to adjust the training progress and intensity, so that better training effect is achieved and the training time is saved.

Description

Technical Field

[0001] This invention mainly relates to a teleoperation system, specifically to a teleoperation intelligent control rehabilitation robot and rehabilitation training method based on digital twin and multi-sensor data fusion. During the training process, it can reproduce the trainer's actions while taking into account the patient's emotions to adjust the training. Background Technology

[0002] For patients with limb dysfunction caused by stroke or accidents, rehabilitation training can effectively restore neuromuscular function and greatly reduce the possibility of lifelong disability. If patients require scientific rehabilitation training, the intervention of a therapist is necessary. However, with the overall increasing prevalence of the disease, the number of therapists and patients is gradually becoming mismatched. Therefore, remotely operated rehabilitation robots have emerged to facilitate one-to-many operation by a therapist.

[0003] Teleoperation systems primarily refer to systems where an operator controls a slave robot to perform exploration and tasks via a master human-machine interface (HMI). A typical teleoperation system consists of an operator, an HMI, a master robot, a communication channel, a slave robot, and the environment. In recent decades, with the rapid development of computer technology, automatic control principles, and sensor application technologies, teleoperated rehabilitation robot systems have become increasingly mature and widespread. However, a series of problems still exist:

[0004] 1. Current wearable systems for capturing human upper limb information provide the functions of each module and the settings for the upper arm, forearm, and hand. They can capture upper limb movement information completely and can be applied to fields such as human-computer interaction and teleoperation. However, they have shortcomings such as unreasonable module settings, inconvenience of wearing, and failure to fully consider the characteristics of the human upper limbs of different genders and ages. They cannot make the system suitable for everyone and have not been generalized to the appropriate population. The robot's degrees of freedom cannot be well matched with the degrees of freedom of the human upper limb.

[0005] 2. Existing control systems obtain relatively little physiological feedback data from the patient and do not place the patient in the control loop, making it difficult to achieve the most comfortable state for the patient; thus, it is difficult to carry out one-to-many rehabilitation for different conditions.

[0006] 3. The rehabilitation process is only operated by the trainer, with little patient data feedback, such as a lack of real-time force feedback; the rehabilitation process still requires the trainer to concentrate, and each patient's condition is different, making it difficult to take care of each patient in a one-to-many process.

[0007] 4. The existing system does not utilize the micro-expression indicator, and cannot promptly ascertain the situation when dealing with certain special groups, such as deaf and mute people.

[0008] 5. All training relies on physical operation, and there is a lack of a virtual system to effectively plan the training program in advance before formal training. Often, if it is found to be unsuitable, the program is changed at the last minute, which cannot effectively improve training efficiency or guarantee training results. Summary of the Invention

[0009] The technical problem to be solved by this invention is to provide a digital twin upper limb rehabilitation training robot and method based on multi-sensor fusion and "patient-doctor in-the-loop" control, taking into account the patient's emotions. It mainly solves the problems of mismatch between the upper limb rehabilitation robot and the human upper limb degrees of freedom, and the failure of patients to achieve better rehabilitation results, while saving trainers more time.

[0010] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0011] A digital twin upper limb rehabilitation training robot that takes into account the patient's emotions is characterized by comprising:

[0012] A remotely operated master robot is used to acquire accurate motion information from the trainer, as well as force feedback, heart rate information, and video information from the patient during the rehabilitation process, enabling the trainer to modify and correct the rehabilitation movements.

[0013] At least one wearable exoskeleton robotic arm acquires the patient's force, heart rate, electromyography signals, and facial expressions to reproduce the trainer's movements and guide the patient to perform the same movements; the length of the link between the shoulder and elbow joints and the length of the link fixed between the elbow and wrist are both set to be adjustable according to the patient's body shape.

[0014] The computer system equipped with a digital twin includes a digital twin model built based on the linkage relationship of the robotic arm, as well as a human-computer interaction module and a voice broadcast module; according to different training stages, it learns the trainer's movements and outputs them to guide the exoskeleton robotic arm to replicate the movements.

[0015] In the above technical solution, a data acquisition module is set up, including a force feedback device, a heart rate acquisition device, an electromyography signal acquisition device, and a visual camera set at least in front of the patient's face, all mounted on the exoskeleton robotic arm. The control module is also connected to the main robot, the exoskeleton robotic arm worn by the patient, and the data acquisition system for signal and command interaction.

[0016] In the above technical solution, the visual camera establishes a three-dimensional model of the patient's face by detecting key points on the patient's face. Based on the multimodal facial images, subjective emotional expression factors that are difficult to perceive accurately are treated as difficult samples. During the training process, more weight is given to difficult samples to complete the recognition of the patient's expression and the perception of emotions.

[0017] In the above technical solution, the recognition of the patient's facial expressions and the perception of emotions are set as the output torque compensation or training intensity adjustment of the associated robotic arm. The patient's emotions are quantified as frustration, excitement and disgust, and correspond to rehabilitation training tasks of "over-challenge", "challenge" and "under-challenge" respectively.

[0018] In the above technical solution, during each rehabilitation session, the output torque of the robotic arm is compensated accordingly based on the patient's emotional state and heart rate physiological information to adjust the training intensity. In the initial and intermediate stages of rehabilitation, the error between the trainer's remote control trajectory and the robotic arm's actual trajectory is compensated for in terms of torque output. Assuming the fused data is K, the trajectory error is Δx, and friction is ignored, then:

[0019]

[0020] α and β were obtained through optimization in order to ensure that the patient's physiological data were in a good state. In the later stage of rehabilitation, β was set to 0.

[0021] α represents the proportion of data fusion that has an effect on the force control of the robotic arm, β represents the magnitude of the effect of trajectory error on the force control of the robotic arm, and when β = 0, it means that the trajectory error has no effect on the magnitude of the differential force of the robotic arm.

[0022] In the above technical solution, the robotic arm is a three-degree-of-freedom robotic arm. Two degrees of freedom are provided at the patient's shoulder joint. The z-axis is established along the direction of the arm with the shoulder joint as the origin, and the x-axis is established along the direction perpendicular to the body and forward. In this right-hand coordinate system, the shoulder joint adopts a pitch angle-yaw angle configuration, and there is one degree of freedom at the elbow joint.

[0023] In the above technical solution, the position of the aforementioned visual camera is adjusted or another visual camera is set up to obtain the trajectory information of the robotic arm's elbow joint and end effector in the world coordinate system.

[0024] In the above technical solution, the LSM algorithm is used to calibrate the camera and complete the transformation of the coordinates of the camera's terminal nodes and endpoints from the camera coordinate system to the world coordinate system during trajectory acquisition.

[0025] In the above technical solution, the digital twin is configured with different modes according to the stage of rehabilitation, including:

[0026] In the initial teaching mode of rehabilitation, while the trainer performs rehabilitation movements for the patient, the digital twin assists the trainer in monitoring whether the patient's various physiological data are normal and learns the trainer's movements.

[0027] When rehabilitation transitions to the second mode in the middle stage, it replaces the trainer in completing the rehabilitation training tasks and autonomously generates control sequences that should have been issued by the master robot, which are then converted into control sequences for the slave robot through the control module.

[0028] The third mode in the later stage of rehabilitation involves collecting sufficient patient data and combining it with the rehabilitation movements learned in the early stages. In a simulation environment, the trained movements are improved to achieve better rehabilitation results. In subsequent rehabilitation, the simulation environment is mapped to the real environment.

[0029] In the above technical solution, the digital twin relies on the patient's physiological information to learn the learned digital twin model. Based on the learned digital twin model, in each mode, the digital twin is set to guide the patient to imitate the trainer's actions through remote operation, and the training difficulty is set to be associated with the patient's facial emotions and physiological information.

[0030] The above-mentioned upper limb rehabilitation teleoperation rehabilitation robot based on "patient-doctor in-loop" control is characterized by: using the Domain Randomization algorithm to map the simulation environment to the real environment, and randomizing the parameters in the simulation environment so that the parameters in the simulation environment largely encompass the parameters in the real environment.

[0031] The upper limb rehabilitation teleoperation rehabilitation robot of the present invention is characterized by: using the DomainRandomization algorithm to map the simulation environment to the real environment, and randomizing the parameters in the simulation environment so that the parameters in the simulation environment largely encompass the real environment.

[0032] This invention also protects a digital twin upper limb rehabilitation training method that takes into account the patient's emotions, characterized by employing the aforementioned digital twin upper limb rehabilitation training robot that considers the patient's emotions; comprising:

[0033] In the initial stage of rehabilitation, the teaching model involves the digital twin assisting the trainer in monitoring the patient's various physiological data to ensure they are normal while the trainer performs rehabilitation movements for the patient, and also learning the trainer's movements.

[0034] When rehabilitation transitions to the second mode in the middle stage, it replaces the trainer in completing the rehabilitation training tasks and autonomously generates control sequences that should have been issued by the master robot, which are then converted into control sequences for the slave robot through the control module.

[0035] The third mode in the later stage of rehabilitation involves collecting sufficient patient data and combining it with the rehabilitation movements learned in the early stages. In a simulation environment, the trained movements are improved to achieve better rehabilitation results. In subsequent rehabilitation, the simulation environment is mapped to the real environment.

[0036] Each mode includes guiding the patient to imitate the trainer's actions through remote operation, as well as constantly observing and evaluating the physiological information of the patient's facial emotions. If any abnormality occurs, necessary measures are taken through the human-computer interaction module and the voice broadcast module.

[0037] In the above method, the observation and evaluation of training using patients' facial emotional expression factors are carried out according to the following steps:

[0038] During rehabilitation training, key facial points are detected to build a three-dimensional facial model of the patient: based on multimodal facial images, subjective emotions that are difficult to perceive accurately are treated as difficult samples, and more weight is given to difficult samples during training; the three-dimensional facial model of the patient is trained and used for rehabilitation training.

[0039] Collect emotion recognition factors during rehabilitation training, and match different task difficulties according to the specific state of the emotion at the time of recognition;

[0040] Set different rules for decreasing or increasing difficulty between different emotions.

[0041] In summary, this invention is a remote-operated rehabilitation robot technology based on digital twins and multi-data fusion. It should consist of four main parts: a master robot operated by a trainer, a wearable exoskeleton slave robot for the patient, a data acquisition system, and a computer system equipped with a digital twin.

[0042] The rehabilitation robot's master end unit's function is to acquire accurate movement information from the trainer and reflect the patient's forces during rehabilitation. Through force feedback, the trainer can modify and correct the rehabilitation movements. The wearable exoskeleton slave end unit's function is to replicate the trainer's movements and guide the patient through the same exercises. The data acquisition system includes visual cameras, electromyography (EMG) sensors, and force feedback devices, using the collected data to evaluate the patient's condition. The computer system equipped with a digital twin primarily uses the trainer as a learning object, with different modes performing different functions.

[0043] This invention has high reliability and provides a new approach to rehabilitation. It can effectively solve the problem of the current imbalance between the number of professional medical personnel and patients, enabling patients to achieve better rehabilitation results while reducing the workload of medical personnel. These advantages make this invention effective in rehabilitation settings.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] Existing collaborative robots for upper limb training rely on a control module that receives online control decisions from an electromyography (EMG) module and motion parameter settings from a software module to control the robotic arm for safe upper limb rehabilitation training. They primarily focus on the acquisition and processing of EMG signals, using EMG signals and collaborative robot technology to assist in upper limb rehabilitation. However, trainers cannot participate in the training process, and there is no data acquisition of facial expressions, making it impossible to simulate a realistic training process.

[0046] This invention employs teleoperation technology, multi-sensor data fusion, and the cooperation of a digital twin. Through teleoperation, patients can mimic the actions of the trainer, facilitating remote rehabilitation, and the trainer's guidance can enhance the rehabilitation effect. Multi-sensor data fusion improves the reliability of patient data feedback, enabling the controller to make more intelligent control. The digital twin plays different functions at different stages of rehabilitation, improving the flexibility and personalization of rehabilitation.

[0047] Secondly, this invention combines digital twin technology to assist trainers in completing different tasks at different stages of rehabilitation. Initially, the digital twin learns the trainer's movements; in the middle stage, the digital twin can guide the patient's movements; in the later stage, after collecting sufficient patient data, the digital twin can simulate in a virtual environment, modifying rehabilitation movements to achieve better rehabilitation results. The digital twin can replace the trainer in performing corresponding movements, mapping from virtual to reality and controlling the movement of the rehabilitation robot.

[0048] Furthermore, this invention integrates multi-sensor body signal data into the master robot, while the slave robot is equipped with a vision camera to capture real-time facial images of the patient. Based on deep learning algorithms, it obtains the patient's emotional information through facial expressions. This emotional information is then rationally fused with data such as heart rate to form a comprehensive picture that reflects the patient's condition to a certain extent. Feedback from various patient data is used to adjust relevant parameters of the controller, ensuring the patient is also included in the system's control loop, thus achieving optimal comfort throughout the rehabilitation process. The control strategy is adjusted according to different rehabilitation stages to achieve better rehabilitation outcomes.

[0049] Secondly, this invention provides a force feedback mechanism to enhance the trainer's operational experience, improve the precision of the operation, and enable patients to achieve better rehabilitation results.

[0050] Finally, the mechanical structure of the slave robot designed in this invention is suitable for a wide range of people. When dealing with people with different upper limb lengths, the robotic arm can be adjusted to fix it in the position that is most comfortable for the patient and provides the best rehabilitation effect. This mechanical mechanism also better conforms to the degree of freedom of the human upper limb, thus achieving better rehabilitation results. Attached Figure Description

[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a block diagram of the upper limb rehabilitation teleoperation robot system based on "patient-doctor in-the-loop" control according to the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the degree of freedom principle of the end-user robot or robotic arm of the present invention.

[0054] Figure 3 This is a schematic diagram of the structure of the robotic arm of the present invention.

[0055] Figure 4 This is a flowchart illustrating the process of emotion recognition-assisted training according to the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0057] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0058] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0059] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this application is in use. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on this application. In addition, the terms "first," "second," and "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0060] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0061] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0062] In this application, unless otherwise expressly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature being directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature being directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0063] The features and performance of this application will be further described in detail below with reference to the embodiments.

[0064] Example 1

[0065] See Figure 1The upper limb rehabilitation teleoperation system based on digital twin and multi-sensor fusion, implemented according to the present invention, includes a master-slave device for a teleoperation robot, a control module, a data acquisition module, and a digital twin. The master-slave device of the teleoperation robot includes a master robot operated by the trainer and a wearable exoskeleton slave robot for the patient.

[0066] The control module connects to the master robot operated by the trainer, the slave robot worn by the patient's exoskeleton, and the data acquisition system via communication links. Different communication links can be selected depending on the specific application environment. In this example, the control module is connected to the master robot via a wired link and to the slave robot and data acquisition module via wireless links. The control module quickly controls the slave robot to move in the same manner as the master robot based on input from the master robot, and corrects the control based on data feedback from the data acquisition module to achieve better rehabilitation results.

[0067] See Figure 1 The data acquisition module consists of a visual camera, an electromyography (EMG) signal acquisition device, a force feedback device, and a heart rate monitor. The visual camera is positioned directly in front of the patient, allowing the trainer to directly observe the patient's condition and collect facial information. It uses deep learning algorithms to evaluate the patient's emotional state during the rehabilitation process, which, along with subsequent physiological information, serves as a standard for evaluating the patient's condition. The EMG signal acquisition device, force feedback device, and heart rate monitor are mounted on the slave robot. As the slave robot follows the master robot, the data acquisition device collects various information and feeds it back to the control system.

[0068] The digital twin primarily uses the trainer as its learning object and has different functions in different modes. Its functions include: (1) While the trainer operates the main end of the rehabilitation robot, the digital twin learns the trainer's actions and the trainer's response to various emergencies. At the same time, relying on the model obtained by learning the patient's physiological information from the deep learning neural network in the early stage, it can observe and evaluate the patient's physiological information at all times during rehabilitation. If an abnormality occurs, it will take necessary measures. (2) It takes over the trainer's task of operating the main end of the rehabilitation robot. With the digital twin's learning from the trainer, it can, to a certain extent, replace the trainer in completing some rehabilitation training tasks. The trainer only needs to observe whether the patient has any abnormalities. (3) Based on a large amount of patient data and some rehabilitation methods of the trainer, the digital twin can learn autonomously in a virtual environment, discover actions and rehabilitation durations with better rehabilitation effects, and after obtaining certification from the trainer, it can be used on the patient.

[0069] Furthermore, the slave robot is a three-degree-of-freedom robotic arm configured with 2-1 degrees of freedom, as shown in the appendix. Figure 2The robotic arm is worn as an exoskeleton on the patient, with two degrees of freedom at the shoulder joint. A z-axis is established along the arm's direction with the shoulder joint as the origin, and an x-axis is established along a direction perpendicular to the body's forward movement. In this right-handed coordinate system, the shoulder joint uses a pitch-yaw configuration. There is also one degree of freedom at the elbow joint. (See appendix.) Figure 3 The length of the link between the shoulder and elbow joints and the length of the link between the elbow and wrist can be adjusted according to the patient. The adjusted length will be fed back to the control module as data for posture calculation.

[0070] The following describes specific embodiments of the present invention.

[0071] The main function of this invention is to generate and send a master-slave operation control sequence in real time, under the joint observation of a digital twin and a trainer, of the trainer's rehabilitation movements, thereby controlling the exoskeleton rehabilitation robot on the patient to complete the specified rehabilitation movements. Addressing some problems mentioned in the background, this invention provides a rehabilitation robot design based on the combination of digital twin and multi-sensor data, tailored to the characteristics of these problems.

[0072] Once the patient's slave robot is activated, the patient's information will appear in the trainer's system. The trainer can choose to connect the master robot to a patient's slave robot or multiple patients' slave robots via a communication link, according to actual needs. One trainer can then guide multiple patients through rehabilitation simultaneously.

[0073] In order to ensure low latency during remote connection, this example uses 5G communication technology as the basis to achieve end-to-end transmission of a large amount of specific data, which almost eliminates the occurrence of erroneous operations caused by latency and improves the stability of the system.

[0074] After a successful communication connection, the patient correctly puts on the exoskeleton end-device robot. Even better, before putting it on, the patient can adjust the length of the link between the shoulder and elbow joints of the robotic arm and the link between the elbow and wrist fixation point according to their own body characteristics. This design provides good support for the patient's upper limbs and avoids secondary injury to the patient during the rehabilitation process.

[0075] The variable link length broadens the user base for robotic arms and saves resources. When wearing the robotic arm, correctly installing data monitoring sensors such as electromyography (EMG) sensors and heart rate monitors ensures data reliability.

[0076] In addition, a visual camera needs to be placed in front of the patient; in the preferred example, a Kinect depth camera was used. In this example, the camera was first calibrated using MATLAB toolboxes, and the camera's depth information was matched with RGB information to obtain the camera's extrinsic parameter matrix, intrinsic parameter matrix, distortion coefficients, etc.

[0077] In a preferred embodiment, this camera primarily performs two functions: acquiring more facial information from the patient, performing 3D reconstruction of the facial information to obtain multidimensional data, and introducing multimodal facial images; and acquiring the trajectory information of the end effector's elbow joint and end effector in the world coordinate system. Alternatively, a separate camera can be used for trajectory information acquisition.

[0078] For Task 1, a 3D facial model of the patient was established by detecting key facial points. Based on multimodal facial images, subjective emotions, which are difficult to perceive accurately, were treated as hard samples. During training, these hard samples were given more weight, thus enabling the recognition of patient expressions and the perception of emotions. The role of emotional value in the entire system is mainly to assess the patient's state and adjust the training intensity accordingly. The process of its function is detailed in the appendix. Figure 4 As shown, it includes the following steps:

[0079] First, the difficulty of the training task is randomly set, and rehabilitation training is carried out based on the currently set difficulty.

[0080] During rehabilitation training, key facial points are detected to build a three-dimensional facial model of the patient: based on multimodal facial images, subjective emotions that are difficult to perceive accurately are treated as difficult samples, and more weight is given to difficult samples during training; the three-dimensional facial model of the patient is trained and used for rehabilitation training.

[0081] Emotion recognition factors during rehabilitation training are collected, and different task difficulties are matched according to the specific state of the recognized emotion. Different rules for decreasing or increasing difficulty are set between different emotions.

[0082] In this example, the patient's emotions are quantified as frustration, excitement, and aversion, corresponding to rehabilitation training tasks of "over-challenge," "challenge," and "under-challenge."

[0083] For task two, to accurately transform the coordinates of the joints and end points from the camera coordinate system to the world coordinate system and reduce errors during camera calibration, the LSM method is used in this example. We assume the end point's coordinates in the robot arm coordinate system are (xr, yr, zr) and in the Kinect camera coordinate system are (xc, yc, zc). Then, the transformation matrix T is such that:

[0084]

[0085] Where (xr,yr,zr) represents the coordinates of the end point in the robot arm coordinate system; (xc,yc,zc) represents the coordinates of the end point in the Kinect camera coordinate system;

[0086]

[0087] According to the LSM algorithm, X cL *T cL =X rL ;

[0088] Where I represents the identity matrix; [xri,yri,zri]i=n, [xci,yci,zci] are the coordinates of the i-th point in the robot coordinate system and the Kinect coordinate system, respectively; Xcl represents the coordinates of the end point in the Kinect camera coordinate system obtained by the LSM algorithm; Xrl represents the coordinates of the end point in the robotic arm coordinate system obtained by the LSM algorithm.

[0089] therefore, Verification has shown that using this algorithm effectively reduces errors caused by camera calibration.

[0090] The control module is responsible for data processing, including data filtering, data fusion, forward and inverse kinematics algorithms for the robotic arm, impedance control calculation, and artificial intelligence optimization algorithms, which transform the collected trainer action information into control data for the slave robot.

[0091] Considering the nonlinear characteristics of the collected data, this example employs an extended Kalman filter for data filtering and fusion, using a first-order Taylor series expansion, as follows: By linearizing the nonlinear system, the data from each sensor can be reasonably fused using a linear Kalman filter.

[0092] In terms of controlling the slave robot, the control module employs different control methods at different stages of rehabilitation. In the early stages of rehabilitation, remote control by the trainer is dominant, with the slave robot providing a larger torque output to drive the patient's movements. In the middle stages of rehabilitation, remote control by the trainer and the patient's voluntary movements become dominant, with the slave robot providing a smaller torque output as the patient's initial voluntary movement strength is relatively low. The slave robot primarily serves as a guide for the patient's movements. In the later stages of rehabilitation, the patient's own movements become the primary focus. At this stage, the patient mainly engages in muscle training, and the slave robot adopts compliant control. When the patient performs movements, the slave robot provides protection. Simultaneously, based on the patient's different movement trajectories, the slave robot adjusts its impedance accordingly to achieve smooth and compliant movements.

[0093] Furthermore, during each rehabilitation session, the output torque of the robotic arm is compensated accordingly based on the patient's emotions, heart rate, and other physiological information to adjust the training intensity. In the initial and intermediate stages of rehabilitation, the error between the trainer's telescopic trajectory and the actual trajectory of the slave robot is compensated for in the torque output. In this example, assuming the fused data is K, the trajectory error is Δx, and friction is ignored, then:

[0094]

[0095] α and β were obtained through optimization in order to ensure that the patient's physiological data were in a good state. In the later stage of rehabilitation, β was set to 0.

[0096] The digital twin in this invention provides multiple functional modes. In an optimized example, during the initial stage of rehabilitation, the digital twin is set to teaching mode. While the trainer performs rehabilitation movements for the patient, the digital twin assists the trainer in monitoring the patient's physiological data and learning the trainer's movements. Through learning the trainer's rehabilitation movements in the early stages, once rehabilitation progresses to the intermediate stage, the digital twin can be switched to a second mode to replace the trainer in completing rehabilitation training tasks. It autonomously generates control sequences that would normally be issued by the master device, which are then converted into slave robot control sequences by the control module. In the later stages of rehabilitation, after collecting sufficient patient data and combining it with the previously learned rehabilitation movements, the trained movements are improved in a simulation environment to achieve better rehabilitation results. In subsequent rehabilitation, the data is mapped to the real environment via sim2real.

[0097] In this example, the Domain Randomization algorithm is used to map the simulation environment to the real environment. The parameters in the simulation environment are randomized, ensuring that the parameters in the simulation environment largely reflect those in the real environment, facilitating direct migration to real-world deployment. The implementation of the digital twin is the focus of this example.

[0098] The embodiments described above are some, but not all, of the embodiments of this application. The detailed description of the embodiments of this application is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

Claims

1. A digital twin upper limb rehabilitation training robot that takes into account the patient's emotions, characterized in that... include: A remotely operated master robot is used to acquire accurate motion information from the trainer and to acquire force feedback, heart rate information, and video information from the patient during the rehabilitation process, enabling the trainer to modify and correct the rehabilitation movements. At least one wearable exoskeleton robotic arm acquires the patient’s force, heart rate, electromyographic signals and facial expressions, respectively, to reproduce the trainer’s movements and guide the patient to perform the same movements. A computer system equipped with a digital twin includes a digital twin model built based on the linkage relationship of the robotic arm, as well as a human-computer interaction module and a voice broadcast module; Depending on the different training stages, learn the trainer's movements and output them to guide the exoskeleton robotic arm to replicate the movements; During each rehabilitation session, the output torque of the robotic arm is compensated accordingly based on the patient's physiological information, including emotions, heart rate, and electromyography, to adjust the intensity of the training. In the early and middle stages of rehabilitation, compensation is introduced for the error between the trainer's telescopic trajectory and the robotic arm's actual trajectory in terms of torque output; assuming the above-mentioned physiological information data fusion data is... K The trajectory error is If we ignore friction, then we have: ； in: Indicates the input torque of the joint; Indicates joint angle; Indicates the second derivative of the joint angle; Indicates the first derivative of the joint angle; This represents the mass matrix or inertia matrix, which is related to the joint angles. This represents the Coriolis force and centripetal force or viscous force generated by motion; This represents the torque distributed across the joints by the robot's own weight under the influence of gravity. The scaling factor represents the fused data. K The degree to which force control is applied; β The proportionality coefficient represents the trajectory error. The degree to which it plays a role in the force control of the robotic arm; β A value of 0 indicates that the trajectory error affects the force control of the robotic arm, particularly in the later stages of rehabilitation. .

2. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 1, characterized in that: The data acquisition module includes a force feedback device, a heart rate acquisition device, an electromyography signal acquisition device mounted on the exoskeleton robotic arm, and a visual camera positioned at least directly in front of the patient's face. The control module is connected to the main robot, the exoskeleton robotic arm worn by the patient, and the data acquisition system for signal and command interaction.

3. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 2, characterized in that: The visual camera establishes a three-dimensional model of the patient's face by detecting key points on the face. Based on multimodal facial images, subjective emotional expression factors that are difficult to perceive accurately are treated as difficult samples. During the training process, more weight is given to difficult samples to complete the recognition of the patient's expression and the perception of emotions.

4. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 3, characterized in that: The recognition of patient facial expressions and the perception of emotions are set as output torque compensation or training intensity adjustment of the associated robotic arm. The patient's emotions are quantified as frustration, excitement and disgust, and correspond to rehabilitation training tasks of "over-challenge", "challenge" and "under-challenge" respectively.

5. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 1, characterized in that: The robotic arm is a three-degree-of-freedom robotic arm. It has two degrees of freedom at the patient's shoulder joint. The z-axis is established along the direction of the arm with the shoulder joint as the origin, and the x-axis is established along the direction perpendicular to the body and forward. In the right-hand coordinate system, the shoulder joint adopts a pitch-yaw angle configuration, and there is one degree of freedom at the elbow joint.

6. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 1, characterized in that: The length of the link between the shoulder and elbow joints and the length of the link between the elbow and wrist are both set to be adjustable according to the patient's body shape.

7. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 1, characterized in that: The LSM algorithm is used to calibrate the camera, and the coordinates of the camera's endpoints and end points are transformed from the camera coordinate system to the world coordinate system during trajectory acquisition.

8. The digital twin upper limb rehabilitation training robot that takes into account the patient's emotions according to claim 1, characterized in that: The digital twin is configured with different modes according to the stage of rehabilitation, including: a teaching mode in the initial stage of rehabilitation, in which the digital twin assists the trainer in monitoring whether the patient's various physiological data are normal while the trainer performs rehabilitation movements for the patient, and learns the trainer's movements. When rehabilitation transitions to the second mode in the middle stage, it replaces the trainer in completing the rehabilitation training tasks and autonomously generates control sequences that should have been issued by the master robot, which are then converted into control sequences for the slave robot through the control module. The third mode in the later stage of rehabilitation involves collecting sufficient patient data and combining it with the rehabilitation movements learned in the early stages. In a simulation environment, the trained movements are improved to achieve better rehabilitation results. In subsequent rehabilitation, the simulation environment is mapped to the real environment. In each mode, the digital twin is set to guide the patient to imitate the trainer's actions through remote operation, and the training difficulty is set to be associated with the patient's facial emotions and physiological information.

9. An upper limb rehabilitation training method that takes into account the patient's emotions, characterized in that... The digital twin upper limb rehabilitation training robot described in any one of claims 1-8 takes into account the patient's emotions; if an abnormality occurs, necessary measures are taken through the human-computer interaction module and the voice broadcast module.

Images