A multimodal neural-driven rehabilitation robot control method and system

Abstract

The embodiment of the application provides a kind of multimodal neural drive's rehabilitation robot control method and system, it is related to the technical field of robot control technology.The method includes: obtaining human-computer interaction task data and user real-time user signal;Based on the user signal and the preset identification system, generate digital signal;According to the difference between the digital signal and simulation signal, or according to the difference between human-computer motion / force interaction data and simulation motion / force data, the preset model corresponding to each is corrected online;Based on the parameters of the model after correction and the historical training data obtained, generate rehabilitation evaluation data;According to the rehabilitation evaluation data, generate task data and training parameters.Through the application, the problem of low training strategy accuracy caused by weak individual adaptive adjustment ability is solved, and the effect of improving individual adaptive adjustment ability is achieved.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the technical field of rehabilitation robot control, in particular to a multi-modal neural driving rehabilitation robot control method and system. BACKGROUND

[0002] Neurological diseases, such as post-stroke hemiplegia, often lead to motor dysfunction in patients, seriously affecting their quality of life, and rehabilitation training is a key link for functional recovery. Traditional rehabilitation training mainly relies on manual assistance by physical therapists or passive traction by mechanical arms, and such methods have the problems of insufficient individualization of training programs and difficulty in dynamically adjusting training strategies according to the real-time neuromuscular state of patients during training, which may lead to mismatched training intensity or the generation of compensatory movement patterns. SUMMARY

[0003] The embodiment of the application provides a multi-modal neural driving rehabilitation robot control method and system to at least solve the problem of low training strategy accuracy in the related art.

[0004] According to an embodiment of the application, a multi-modal neural driving rehabilitation robot control method is provided, comprising:

[0005] Obtaining human-computer interaction task data and real-time user signals of a user, wherein the user signals include electroencephalogram signals, electromyogram signals, human-computer motion / force interaction data, and the human-computer interaction task data includes a task type, a task target and user operation feedback;

[0006] Generating digital signals based on the user signals and a preset recognition system, wherein the digital signals include digital motion signals and digital activation signals; and generating simulation signals based on a preset model and the human-computer interaction task data, wherein the simulation signals include simulation motion signals, simulation activation signals and simulation motion / force data;

[0007] Online correcting the respective preset models according to the difference between the digital signals and the simulation signals, or according to the difference between the human-computer motion / force interaction data and the simulation motion / force data;

[0008] Generating rehabilitation evaluation data based on the parameters of the corrected model and the obtained historical training data;

[0009] Generating task data and training parameters according to the rehabilitation evaluation data, wherein the task data is used for task information interaction between a rehabilitation robot and a user, and the training parameters are used for motion / force interaction between the rehabilitation robot and the user.

[0010] In one example embodiment, the difference between the digital signal and the simulated signal comprises:

[0011] comparing the digital motion signal with a simulated motion signal outputted from the cognitive-motor model based on the human-machine interaction task data, obtaining a first difference, and using a first machine learning algorithm to correct the cognitive-motor model online according to the first difference, wherein the preset model comprises the cognitive-motor model;

[0012] comparing the digital activation signal with a simulated activation signal outputted from the skeletal muscle control model based on the digital motion signal, obtaining a second difference, and using a second machine learning algorithm to correct the skeletal muscle control model online according to the second difference, wherein the preset model comprises the skeletal muscle control model;

[0013] correcting the preset model online according to the difference between the human-machine motion / force interaction data and the simulated motion / force data comprises:

[0014] comparing the real-time human-machine motion / force interaction data with simulated motion / force data outputted from the musculoskeletal model based on the simulated activation signal, obtaining a third difference, and using a third machine learning algorithm to correct the musculoskeletal model online according to the third difference, wherein the preset model comprises the musculoskeletal model.

[0015] In one example embodiment, the generating a digital signal based on the user signal and a preset recognition system comprises:

[0016] generating a digital motion signal representing a user's motion intention based on the electroencephalogram signal and a preset motion signal recognition system;

[0017] generating a digital activation signal representing a muscle activation state based on the electromyogram signal and a preset activation signal recognition system.

[0018] In one example embodiment, the first machine learning algorithm, the second machine learning algorithm, and the third machine learning algorithm each comprise a recurrent neural network, a long short-term memory network, a Gaussian process regression, a reinforcement learning policy gradient method, or an online adaptive filter.

[0019] In one example embodiment, after the preset model is corrected online, the method further comprises:

[0020] periodically obtaining a model snapshot of the corrected preset model;

[0021] generating predicted physiological response data according to the model snapshot;

[0022] comparing the predicted physiological response data with preset reference response data, calculating an evaluation index of the model snapshot;

[0023] adjusting a meta-parameter according to the evaluation index, wherein the meta-parameter is used for online adjustment of the preset model.

[0024] According to another embodiment of the present application, a multi-modal neural-driven rehabilitation robot control system is provided, comprising:

[0025] a data acquisition module, configured to acquire human-computer interaction task data and real-time user signals of a user, wherein the user signals comprise electroencephalogram signals, electromyogram signals, human-computer motion / force interaction data, and the human-computer interaction task data comprises a task type, a task target and user operation feedback;

[0026] a signal generation module, configured to generate digital signals based on the user signals and a preset recognition system, wherein the digital signals comprise digital motion signals and digital activation signals, and to generate simulation signals based on a preset model and the human-computer interaction task data, wherein the simulation signals comprise simulation motion signals, simulation activation signals and simulation motion / force data;

[0027] a correction module, configured to correct the respective preset model online according to a difference between the digital signals and the simulation signals, or according to a difference between the human-computer motion / force interaction data and the simulation motion / force data;

[0028] a rehabilitation evaluation module, configured to generate rehabilitation evaluation data based on parameters of the corrected model and acquired historical training data;

[0029] a task generation module, configured to generate task data and training parameters according to the rehabilitation evaluation data, wherein the task data is used for task information interaction between a rehabilitation robot and a user, and the training parameters are used for motion / force interaction between the rehabilitation robot and the user.

[0030] In one exemplary embodiment, the difference between the digital signals and the simulation signals comprises:

[0031] comparing the digital motion signals with simulation motion signals output by a cognitive-motion model based on the human-computer interaction task data to obtain a first difference, and correcting the cognitive-motion model online according to the first difference by using a first machine learning algorithm, wherein the preset model comprises the cognitive-motion model;

[0032] comparing the digital activation signal with a simulation activation signal output based on a preset skeletal muscle control model according to the digital motion signal, obtaining a second difference value, and using a second machine learning algorithm to correct the skeletal muscle control model online according to the second difference value, wherein the preset model includes the skeletal muscle control model;

[0033] correcting the preset model online according to the difference between the human-computer motion / force interaction data and the simulation motion / force data includes:

[0034] comparing the real-time human-computer motion / force interaction data with simulation motion / force data output based on a musculoskeletal model according to the simulation activation signal, obtaining a third difference value, and using a third machine learning algorithm to correct the musculoskeletal model online according to the third difference value, wherein the preset model includes the musculoskeletal model.

[0035] In one exemplary embodiment, the generating of the digital signal based on the user signal and a preset recognition system includes:

[0036] generating a digital motion signal representing the user's motion intention based on the electroencephalogram signal and a preset motion signal recognition system;

[0037] generating a digital activation signal representing the muscle activation state based on the electromyogram signal and a preset activation signal recognition system.

[0038] According to another embodiment of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.

[0039] According to another embodiment of the present application, an electronic device is also provided, which includes a memory and a processor, the memory stores a computer program, and the processor is configured to execute the computer program to perform the steps in any of the above method embodiments.

[0040] The present application can capture the initiative of the user from the neural intention level by constructing a digital twin model that is real-time synchronized with the real user and bidirectionally interacts, and can make the digital twin approach the real physiological state of the user through continuous machine learning correction, thereby generating a highly personalized and adaptive training strategy, solving the technical problem of low training strategy precision caused by insufficient personalization in traditional rehabilitation training, and achieving the beneficial effects of improving rehabilitation effect and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0041] Figure 1is a structural block diagram of a multi-modal neural driven rehabilitation robot control system according to an embodiment of the present application;

[0042] Figure 2 is a control flow chart according to a specific embodiment of the present application. DETAILED DESCRIPTION

[0043] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0044] Hereinafter, the terms "first", "second", and the like are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second", and the like can explicitly or implicitly include one or more features. In the description of the present application, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0045] In addition, in the present application, the orientation terms such as "up", "down", "left", "right", etc. can include but not limited to the orientation defined by the relative position of the components in the drawings. It should be understood that these directional terms can be relative concepts, which are used for relative description and clarification, and can be changed accordingly according to the change of the position of the components in the drawings.

[0046] In the present application, unless otherwise specified and limited, the term "connection" should be understood broadly, for example, "connection" can be fixed connection, or detachable connection, or integral; can be directly connected, or indirectly connected through intermediate medium. In addition, the term "coupling" can be an electrically connected manner for signal transmission.

[0047] As used herein, "about", "approximately" or "approximately" includes the stated value and the average value within the acceptable deviation range of the specific value, wherein the acceptable deviation range is determined by the ordinary skill in the art considering the measurement being discussed and the error related to the measurement of the specific quantity (i.e. the limitation of the measurement system).

[0048] As Figure 1 As shown in the figure, the present application provides a functional block diagram of a multi-modal neural driven rehabilitation robot control system, the system can include a data acquisition module 100, a signal generation module 200, a correction module 300, a rehabilitation evaluation module 400 and a task generation module 500, these modules can be connected through bus or other ways, realize the communication between each other, wherein:

[0049] The data acquisition module 100 is configured to acquire human-computer interaction task data and real-time user signals of a user, wherein the user signals include electroencephalogram signals, electromyogram signals, and human-computer motion / force interaction data, and the human-computer interaction task data includes a task type, a task target, and user operation feedback.

[0050] The signal generation module 200 is configured to generate digital signals based on the user signals and a preset recognition system, wherein the digital signals include digital motion signals and digital activation signals; and generate simulation signals based on a preset model and the human-computer interaction task data, wherein the simulation signals include simulation motion signals, simulation activation signals, and simulation motion / force data.

[0051] The correction module 300 is configured to correct the respective preset model based on a difference between the digital signals and the simulation signals, or based on a difference between the human-computer motion / force interaction data and the simulation motion / force data.

[0052] The rehabilitation evaluation module 400 is configured to generate rehabilitation evaluation data based on parameters of the corrected model and acquired historical training data.

[0053] The task generation module 500 is configured to generate task data and training parameters based on the rehabilitation evaluation data, wherein the task data is used for task information interaction between a rehabilitation robot and a user, and the training parameters are used for motion / force interaction between the rehabilitation robot and the user.

[0054] Reference Figure 2 It shows the overall architecture of the multi-modal information fusion human motion function digital twin and rehabilitation robot closed-loop control system described in the embodiments of the application. The system mainly consists of three interacting core parts: a real user, a rehabilitation robot, and a digital user (i.e., a digital twin) as a bridge between the two.

[0055] The real user is the starting point and the end point of the entire closed loop, and its biological structure includes the cognitive center, the motor center, the spinal cord, the skeletal muscle and the skeleton. The system monitors the physiological activity of the user in real time through multi-modal sensors.

[0056] The rehabilitation robot is a physical entity for execution and interaction, and its internal functional modules include a vision system, an evaluation system, a scheme system, a training system and a motion system.

[0057] The digital user (digital twin) is a high-fidelity virtual model of the real user's neural-muscular-skeletal system constructed in a computing environment. It contains three core level models: a motor nerve model (i.e., a cognitive-motor model), a skeletal muscle control model and a muscle-skeletal simulation system (i.e., a muscle-skeletal model).

[0058] The information flow and control flow among the three constitute the core working logic of the system. The real user generates a motion intention, which is captured through electroencephalogram (EEG) signals. At the same time, the actual activation state of the muscle is collected through surface electromyography (sEMG) signals. The actual limb movement of the user and the interaction force / displacement generated by the rehabilitation robot are captured by the motion sensor. These multi-modal data streams from the real user are input into the evaluation system of the rehabilitation robot on the one hand, for generating macro training targets; on the other hand, they are used as “real labels” for comparison and correction with the simulation output of the digital twin model.

[0059] The digital twin model receives the motion / force target from the rehabilitation robot as input instructions and runs simulation. The motor neural model simulates the activity of the cerebral cortex to generate a “digital motion intention”. The skeletal muscle control model converts this intention into a control signal for the virtual muscle, that is, a “digital motion signal”. The musculoskeletal simulation system calculates the motion trajectory of the virtual limb and the force interacting with the outside world according to these control signals, that is, “digital motion / force simulation data”.

[0060] The system continuously compares the “digital motion intention” with the intention decoded from the real EEG, compares the “digital motion signal” with the signal collected from the real sEMG, and compares the “digital motion / force simulation data” with the data captured by the real sensor. The differences or errors generated by these comparisons are input into the respective corresponding machine learning algorithms for online and continuous adjustment and optimization of the internal parameters of the three models in the digital twin. This process enables the digital twin to continuously “learn” and “adapt”, thereby more and more accurately reflecting the individual characteristics and state changes of the real user.

[0061] The corrected and highly personalized digital twin provides a basis for decision-making for the rehabilitation robot. The evaluation system of the rehabilitation robot comprehensively analyzes the signals from the real user and the state of the digital twin to generate a quantitative rehabilitation evaluation report. Based on this evaluation, the scheme system formulates a personalized training scheme for the next stage (for example, adjusting the task difficulty, the assistance force, etc.). The training system converts the scheme into specific instructions to drive the motion system to physically interact with the real user. The response of the user is captured by the sensor, starting a new round of “perception-modeling-simulation-comparison-correction-decision-execution” closed loop, realizing the adaptive rehabilitation training of brain-muscle-robot collaboration.

[0062] The following will be combined with Figure 2 A multi-modal information fusion human motion function digital twin and rehabilitation robot closed loop control method provided by the embodiment of the application will be described in detail.

[0063] S100: Real-time acquisition of user signals in the rehabilitation training process, including real-time electroencephalogram signals, real-time electromyogram signals, and real-time human-machine motion and force interaction data; and acquisition of human-computer interaction task data, including task type, task target, and user operation feedback.

[0064] This step is performed by an integrated multi-modal data acquisition module. The module includes but is not limited to the following devices: Electroencephalogram (EEG) acquisition device: usually a top equipped with multiple (e.g., 32 or 64) electrodes. The electrodes are arranged on the user's scalp surface according to the international 10-20 system standard, for non-invasive recording of the electrical activity of the cerebral cortex; the acquired user signal is a microvolt-level voltage time series reflecting the synchronous activity of large-scale neuron groups. In order to enhance the capture of blood oxygen changes in the brain area related to motor intention, the device can optionally integrate near-infrared spectroscopy (fNIRS) probes, synchronously acquired with the electroencephalogram signal, providing more rich neural activity information.

[0065] Among them, electromyogram (sEMG) acquisition device: usually a set or an array of surface electromyography electrodes, pasted on the muscle belly skin of the main muscle groups (agonist, antagonist, synergist) related to the training task. For example, when performing elbow flexion and extension training of the upper limb, electrodes are placed on the biceps brachii and triceps brachii. The acquired signal is a millivolt-level voltage time series reflecting the sum of motor unit action potentials when muscle fibers contract.

[0066] Human-machine motion and force interaction data acquisition device: this part of data is provided by sensors integrated with the rehabilitation robot. For example, a six-axis force / torque sensor installed on the end effector of the rehabilitation robot, for real-time measurement of interactive forces such as push, pull, and twist applied by the user on the robot; at the same time, the joint encoder of the robot or the external optical / inertial motion capture system (such as VICON or Xsens), for accurately recording the kinematics data of the user's limbs, including joint angle, angular velocity, position and attitude of the limb end, etc.

[0067] The system provides a unified time reference for all acquisition devices through a master clock signal, ensuring that each frame of acquired electroencephalogram data, electromyogram data, and motion / force data is accurately time-stamped, so that accurate time alignment can be performed in subsequent analysis.

[0068] Human-computer interaction task data is used to provide context information for rehabilitation training tasks. These information provide prior knowledge for the model on "what is the task" and "where is the target", making the model's simulation behavior have a clear purpose; human-computer interaction task data mainly includes three aspects:

[0069] Task type: Used to define the nature of the current rehabilitation training; for example, the task can be "trajectory tracking", "goal arrival", "rhythmic movement" or "functional task simulation" (such as drinking water, combing hair); each task type corresponds to different movement patterns and control strategies.

[0070] Task objective: For a "target arrival" task, the task objective may be a coordinate point in three-dimensional space. For a "trajectory tracking" task, the objective might be a predefined spatial trajectory function. For "rhythmic movement," the task objective might be the desired movement frequency. and amplitude .

[0071] User operation feedback: This part of the data records the user's active control over the task process through interactive devices (such as buttons, touch screens, voice commands). For example, the user can issue commands such as "start task", "pause task" or "confirm completion" through buttons.

[0072] For example, suppose a user is performing a "desktop retrieval" task guided by a rehabilitation robot, at a certain point in time... The multimodal data acquisition module simultaneously recorded the following data:

[0073] One dimension is EEG data matrix Where 64 is the number of electrode channels. It is the number of sampling points within a short time window, and the data unit is microvolts (µV). ).

[0074] One dimension is electromyography data matrix Where 8 represents the number of channels in the target muscle, and the data unit is millivolts (mV). ).

[0075] A 6-dimensional force / torque vector The force is measured by a force sensor. The unit of force is Newton (N), and the unit of torque is Newton-meter (N·m).

[0076] A kinematic vector containing the angles of the shoulder, elbow, and wrist joints. The angle is measured by an encoder or motion capture system, and the unit of angle is radians (rad).

[0077] All of this data This constitutes the time A snapshot of the complete state of the system. Throughout the entire rehabilitation training, the system continuously performs this step at a high sampling rate (e.g. 1000 Hz for EEG / sEMG, 100 Hz for motion / force data) to generate a continuous data stream, which provides the input for all subsequent processing steps.

[0078] Before the task starts, the visual system of the rehabilitation robot (or set by the therapist) presents a target water cup in the virtual reality or augmented reality interface. At this moment, the human-robot interaction task data is defined as follows:

[0079] Task type: Task_Type = 'Target_Reaching'.

[0080] Task target: The coordinates of the water cup in the workspace of the robot are determined as (meters), which is subsequently passed to the cognitive-motor model.

[0081] User operation feedback: When the user is ready to start the task, a hand-held button is pressed to confirm, and then the system records an event User_Feedback = 'Start_Signal' with a timestamp This signal can be used as a trigger to start the simulation of the motor intention by the cognitive-motor model.

[0082] These context information makes the simulation of digital twin purposeful, for example, after receiving the task target , the motion planning module in the cognitive-motor model can start to calculate the optimized motion trajectory from the current hand position to the target position, and generate a highly task-related simulation of motor intention, and so on.

[0083] S200: Based on the user signal and the preset recognition system, a digital signal is generated, wherein the digital signal includes a digital motion signal and a digital activation signal; and based on the preset model and the human-robot interaction task data, a simulation signal is generated, wherein the simulation signal includes a simulation motion signal, a simulation activation signal, and simulation motion / force data;

[0084] In this embodiment, the following sub-steps are specifically included:

[0085] S210, based on the real-time electroencephalogram signal and the preset motion signal recognition system, a digital motion signal representing the user's motor intention is generated.

[0086] From the complex, noisy raw EEG signal, the user's "high-level command" about movement, i.e. the user's movement intention, is decoded, which can be a decision about "when to move", "which direction to move" or "how to move"; the decoding result is quantized as a "digital movement signal" and input into the subsequent digital twin model as a "real label" compared with the model simulation result. The movement signal recognition system is a pre-trained signal processing and pattern recognition model. Its processing flow usually includes the following stages:

[0087] Preprocessing: filtering the raw EEG signal to remove noise and artifacts. Common processing includes:

[0088] Band-pass filtering: filtering out frequency components unrelated to neural activity. For example, apply a 0.5-40Hz band-pass filter to retain most of the movement-related EEG rhythms (such as μ rhythm, β rhythm), while removing DC drift and high-frequency noise.

[0089] Notch filtering: filter out 50Hz or 60Hz power frequency interference.

[0090] Spatial filtering: use information from multiple electrode channels to enhance signal-to-noise ratio, such as Laplace filtering or independent component analysis (ICA) to remove eye movement, muscle activity, etc.

[0091] Feature extraction: extract features from the preprocessed signal that can effectively distinguish different movement intentions. Common features include:

[0092] Movement-related cortical potential: about 1-2 seconds before voluntary movement, a slow negative potential drift can be observed in the central motor cortex. Its amplitude, slope and latency can be used as features of movement intention.

[0093] Event-related desynchronization / synchronization (ERD / ERS): during motor imagination or actual movement, the energy of μ rhythm (8-13Hz) and β rhythm (14-30Hz) in the sensorimotor cortex area will decrease significantly (ERD) or rebound after movement (ERS). Different limb or different direction of movement will correspond to specific ERD / ERS patterns in different brain areas, which can be extracted using short-time Fourier transform, wavelet transform or common spatial pattern (CSP) algorithm.

[0094] Pattern classification: input the extracted features into the classifier to determine the user's specific movement intention. Common classifiers include linear discriminant analysis (LDA), support vector machine (SVM) or deep neural network (such as convolutional neural network CNN).

[0095] Output "digital movement signal" is a time series vector, whose specific form depends on the complexity of the decoding task.

[0096] Exemplarily, in a simple left-right hand motor imagery task, the goal of the recognition system is to determine whether the user intends to move the left hand or the right hand; after pre-processing, the system computes the energy of the mu rhythm on the central electrodes C3 and C4; through the CSP algorithm, the optimal spatial filter is found to maximize the variance difference of the left-right hand imagery task in the mu band, and the CSP feature vector is obtained; then a trained LDA classifier receives the CSP feature vector and outputs a probability value to represent the time point The probability of the user's intention to move the right hand.

[0097] The digital motor signal can be represented as:

[0098]

[0099] Suppose at a certain time , the classifier outputs , the system determines that the user currently has a strong intention to move the right hand. This vector is the digital motor signal at that moment, which will be sent to the cognitive-motor model for comparison. This step realizes the conversion from abstract neural activity to specific, quantified motor instructions, which is the first key interface connecting the real user's brain to the digital twin world.

[0100] S220: Based on real-time electromyography signals and a preset activation signal recognition system, a digital activation signal representing the muscle activation state is generated.

[0101] Here, the original, oscillating surface electromyography signal is converted into a smooth signal that quantifies muscle activation level or contraction strength, i.e., the "digital activation signal", which is used to represent muscle-level information and will also serve as the "real label" for comparison with the simulation results of the digital twin model. The processing flow of the activation signal recognition system usually includes:

[0102] Pre-processing: Similar to EEG, the original sEMG signal also needs to be filtered. A band-pass filter (e.g., 20-450 Hz) is usually used to retain the main sEMG spectrum and remove motion artifacts and high-frequency noise; a notch filter is also needed to remove power frequency interference.

[0103] Signal processing: The processed sEMG signal is converted into a measure of activation level, including:

[0104] Rectification: Take the absolute value of all negative values of the signal to obtain the instantaneous energy of the signal.

[0105] Smooth / Envelope extraction: The rectified signal is low-pass filtered (e.g. a Butterworth filter with a cutoff frequency of 2-10 Hz) or a moving average window is applied to obtain a smooth envelope. This envelope reflects the trend of muscle activation level over time.

[0106] Normalization: In order to compare between different muscles, different users, or different measurements, the sEMG envelope needs to be normalized. A common method is to divide by the amplitude of sEMG measured at maximum voluntary contraction (MVC) of the muscle; the normalized signal value is usually between 0 and 1 (or higher, indicating supermaximal contraction), which can be directly interpreted as a percentage of muscle activation level. In addition, the firing time and frequency of individual motor units can be identified from the sEMG signal, providing more detailed neural drive information.

[0107] Output "digital activation signal" is a vector time series with dimensions equal to the number of monitored muscle channels.

[0108] Exemplarily, in an elbow flexion-extension training, the system monitors the sEMG of biceps and triceps.

[0109] At time , the raw sEMG signal collected is preprocessed, full-wave rectified, and low-pass filtered with a 4th order, 6 Hz cutoff frequency, to obtain the envelope value of biceps and the envelope value of triceps .

[0110] Suppose the pre-measured sEMG envelope amplitudes corresponding to MVC of this user are and .

[0111] After normalization, the digital activation signal vector at this time is obtained:

[0112] .

[0113] This vector indicates that at time , the user's biceps activation level is about 30% of its maximum capacity, while the triceps activation level is about 12%, and so on.

[0114] S300: According to the difference between the digital signal and the simulation signal, the respective pre-set model is corrected online.

[0115] This step specifically includes the following sub-steps:

[0116] S310: compare the digital motor signal with the simulated motor signal output by the cognitive motor model based on the human-computer interaction task data, obtain a first difference value, and use a first machine learning algorithm to correct the cognitive-motor model online according to the first difference value.

[0117] This step uses the motor intention decoded from the real brain (digital motor signal) to calibrate the cognitive-motor model that simulates the behavior of the brain.

[0118] The cognitive-motor model is used to simulate the neural computing process from receiving task goals to generating internal motor intentions, which can be a model based on optimal control theory or a complex deep learning model (such as a recurrent neural network RNN or its variant LSTM), the input of which is the acquired human-computer interaction task data (such as target points ), and the output is a "simulated motor signal" The format of this simulated signal is exactly the same as the aforementioned generated digital motor signal , so as to facilitate direct comparison.

[0119] The correction process is as follows:

[0120] Simulation execution: after the cognitive-motor model receives the task goal, it starts internal motor planning and decision simulation and outputs a simulated motor signal sequence over time.

[0121] Comparison and difference calculation: at each time step , the system compares the model's simulation output with the decoded signal from the real EEG , thereby obtaining a first difference value , which can be a simple vector subtraction or a more complex distance metric such as cross-entropy loss (if the signal is a probability distribution):

[0122]

[0123] Online correction: the first machine learning algorithm receives this difference signal , then adjusts the internal parameters of the cognitive-motor model (for example, the weights of the neural network ), so that the output of the next simulation is closer to , thereby reducing the difference. This process usually uses gradient descent type algorithms:

[0124]

[0125] where is the learning rate, and L is the loss function defined based on the difference.

[0126] Exemplarily, for a left-right hand recognition task, the cognitive-motor model employs a small LSTM network, taking as input a task instruction (e.g., "prepare to move right"), and outputs a two-dimensional probability vector:

[0127]

[0128] At time , the true EEG decoding result is .

[0129] Meanwhile, the cognitive-motor model, based on the task instruction, outputs a simulation result under its current parameters , the model "thinks" that the user intends to move the right hand with a probability of 70%.

[0130] The first difference vector is thus: This difference indicates that the model underestimates the user's intention to move the right hand.

[0131] Subsequently, the first machine learning algorithm (e.g., an online backpropagation algorithm) uses this difference to calculate the gradient of the loss function (e.g., mean squared error ), and updates the weights of the LSTM network accordingly; after multiple iterations of correction, the output of the model will gradually converge to and so on.

[0132] S320: Compare the digital activation signal with the simulation activation signal output by the skeletal muscle control model based on the digital motion signal, obtain a second difference, and use the second machine learning algorithm to correct the skeletal muscle control model online based on the second difference.

[0133] This step uses the activation level extracted from the real muscle sEMG (digital activation signal) to "calibrate" the skeletal muscle control model that simulates the conversion process of neural signals to muscle activation. The skeletal muscle control model receives the output of the upper cognitive-motor model (or directly uses the real decoded digital motion signal to reduce error accumulation) as input, and converts this high-level motion instruction into fine activation instructions for each relevant muscle; this model can be a mapping network based on muscle synergy theory, or a complex neural network whose output is a "simulation activation signal" , which is consistent in format with the aforementioned generated digital activation signal .

[0134] The correction process includes:

[0135] Simulation execution: the skeletal muscle control model receives the input motion intention, calculates the theoretical activation level of each muscle, and outputs a simulation activation signal sequence .

[0136] Comparison and difference calculation: at each time step , the system compares the simulation output of the model to the real sEMG extracted from the user , calculates a second difference :

[0137]

[0138] Online correction: the second machine learning algorithm uses the difference to adjust the internal parameters of the skeletal muscle control model

[0139] to minimize the gap between simulated activation and real activation:

[0140]

[0141] Exemplarily, taking the elbow flexion task, the skeletal muscle control model adopts a feedforward neural network, whose input is a scalar representing the desired elbow flexion torque (generated by the upper model according to the task requirements), and the output is a two-dimensional activation vector .

[0142] At time , the upper model instructs that an elbow flexion action is needed, and the skeletal muscle control model calculates the simulated activation signal as according to its current parameters; the model believes that in order to complete the task, the biceps needs 45% activation, and the triceps needs 5% activation to maintain joint stability.

[0143] However, the digital activation signal measured from real sEMG is , and the second difference vector is , which indicates that the model overestimates the biceps activation required to complete the task (possibly because the patient's biceps is more efficient, or there are other synergist muscles compensating), and underestimates the co-activation level of the antagonist muscle (triceps) (which is common in stroke patients, showing poor muscle control coordination).

[0144] Subsequently, the second machine learning algorithm uses this difference to update the weights of the neural network, and through continuous correction, the model will learn the patient's unique muscle recruitment pattern and synergistic strategy, for example, to reduce the dependence on the biceps and appropriately increase the activation of the triceps when generating the same torque, so that the digital twin muscle control strategy is more in line with the user's real physiological conditions.

[0145] S330: compare the real-time human-machine motion and force interaction data with the simulation motion and force data output by the musculoskeletal model based on the simulation activation signal, obtain a third difference value, and use a third machine learning algorithm to correct the musculoskeletal model online according to the third difference value.

[0146] This step is to calibrate the musculoskeletal model simulating the biomechanical response of the limb using the limb motion and interaction force measured from the real sensor (real-time human-machine motion and force interaction data); the musculoskeletal model (also known as the musculoskeletal simulation system) is a complex model based on multi-rigid-body dynamics and biomechanics principles (such as the model built in OpenSim platform), which receives the simulation activation signal output by the upper-layer skeletal muscle control model as input, which contains the geometry of the skeleton, mass inertia parameters, joint motion constraints, and muscle origin and insertion points, force-length-velocity relationship (such as Hill-type muscle model) etc.; according to the input muscle activation signal, the model simulates the kinematics (joint angle ) and dynamics (interaction force with the environment ) of the limb through forward dynamics calculation, and the output is the "simulation motion and force data".

[0147] The correction process is as follows:

[0148] Simulation execution: the musculoskeletal model receives the simulation activation signal, runs forward dynamics simulation, and outputs simulation motion and force data .

[0149] Comparison and difference calculation: at each time step , the system compares the model's simulation output with the motion and force data measured by the real sensor in S100 , and calculates the third difference value :

[0150] Online correction: the third machine learning algorithm uses this difference value to adjust the internal parameters of the musculoskeletal model , which may include the maximum isometric contraction force of the muscle, the tendon relaxation length, the mass or center of mass position of the bone segment, etc.:

[0151]

[0152] Exemplarily, in the elbow flexion and extension task, the rehabilitation robot applies a constant resistance; at time , the simulation activation signal output by the skeletal muscle control model is , then the musculoskeletal model receives this signal and calculates the joint angle ), the angular acceleration of the elbow joint is calculated by simulation, and the simulation joint angle is obtained by integration rad, and the simulation force of the end interacting with the robot .

[0153] However, the real sensor measured data is: the actual joint angle rad, the actual interaction force , and the third difference is , which shows that at the same muscle activation level, the simulation motion of the model (larger angle and larger force) is more forceful than the real user's motion, which may mean that the maximum strength of the biceps brachii set in the model is too high for this user (for example, due to muscle atrophy).

[0154] Subsequently, a third machine learning algorithm (for example, a Kalman filter or a gradient descent optimizer) will make a negative adjustment to according to this error, for example, update it to 580N; in this way, the musculoskeletal model can continuously adjust its biomechanical parameters to match the user's actual physical condition, such as muscle strength, joint stiffness, etc., to build a truly personalized and high-fidelity biomechanical digital twin.

[0155] S400: Obtain the corrected cognitive-motor digital model parameters, skeletal muscle control digital model parameters, and musculoskeletal digital model parameters, as well as historical training data; and based on these data, generate the user's rehabilitation evaluation data through the rehabilitation robot evaluation system.

[0156] After continuous online correction, the three levels of model parameters of the digital twin can accurately reflect the current neural-muscle-skeletal state of the user, and at this time, these model parameters are used to conduct rehabilitation evaluation in combination with long-term training data; The data obtained includes:

[0157] Corrected model parameters:

[0158] Cognitive-motor model parameters : used to indicate the user's reaction time, decision strategy, motion planning ability, and other cognitive level features.

[0159] Skeletal muscle control model parameters : used to indicate the user's muscle coordination mode, activation efficiency, co-activation level, and other neural control level features.

[0160] Musculoskeletal model parameters : used to indicate the user's muscle strength, joint range of motion, tissue viscoelasticity, and other biomechanical level features.

[0161] Historical training data: includes all the training records of the user in the past period of time, such as task completion time, success rate, trajectory tracking error, generated force, sEMG integral value, etc.

[0162] The evaluation system calculates rehabilitation evaluation indicators based on these data:

[0163] Motor function score: functional score obtained by running standard clinical evaluation tasks (such as virtual version of Fugl-Meyer assessment) in digital twin model.

[0164] Muscle activation symmetry index: for hemiplegic patients, the activation patterns of affected and healthy muscles (or compared with standard database) can be compared; for example, the activation ratio of biceps brachii and triceps brachii during training is calculated and compared with normal pattern to quantify coordination disorder.

[0165] Task completion efficiency: assess the economy of user movement by analyzing energy consumption in digital twin model.

[0166] Neural plasticity indicators: quantify changes in brain cortical representation by tracking changes in cognitive-motor model parameters over time; for example, if the model shows that the activation area required to complete the same task becomes closer to the normal pattern, it may be direct evidence of neural remodeling.

[0167] For example, the evaluation system finds that in the past week of training, the maximum strength parameter of the biceps brachii in the user's musculoskeletal model increased from 450N to 480N, while the biceps brachii co-activation parameter in the skeletal muscle control model decreased from 30% to 18% to complete the standard elbow flexion task. Historical data shows that task completion time has been shortened by 20%. Based on these data, the system generates a rehabilitation evaluation data report: "user muscle strength increased by about 6.7%, muscle control coordination improved, movement efficiency increased; suggest entering the next stage, appropriately increase resistance training" and so on.

[0168] S500: based on the rehabilitation evaluation data, generate personalized rehabilitation training program through rehabilitation robot program system; and drive rehabilitation robot to execute, forming a closed loop control.

[0169] This step converts all the results of the previous perception, modeling, simulation, correction and evaluation into actual rehabilitation robot actions. This step is completed by the rehabilitation robot program system, training system and motion system.

[0170] Program generation: the rehabilitation robot program system receives the rehabilitation evaluation data generated as described above, and the system contains an expert knowledge base or a reinforcement learning decision model inside, and then automatically generates or adjusts the rehabilitation training program for the next stage according to the evaluation report.

[0171] Training intensity: If the assessment shows increased muscle strength, the regimen system can increase the resistance applied by the rehabilitation robot in the movement.

[0172] Training frequency: If the assessment shows that the user is fatigued (e.g., sEMG spectrum moves towards low frequencies), the system can recommend reducing training frequency or increasing rest time.

[0173] Task difficulty level: If the assessment shows improved motor coordination, the system can introduce more complex tasks, such as curved trajectory tracking in three-dimensional space, or require faster movement speeds.

[0174] Assistance / resistance mode: The system can dynamically switch modes based on the user's real-time performance. For example, provide an "assisted start" mode when the user has difficulty initiating the movement, and switch to an "antagonist training" mode to strengthen the muscles once the user can actively exert force.

[0175] Execution and interaction:

[0176] 1. The training system, according to the newly generated regimen, decomposes it into specific task data (such as new target points, new trajectories) and training parameters (such as resistance size , assistance force threshold).

[0177] 2. The task data is presented to the user through the interaction system (such as screen display), achieving task information interaction.

[0178] 3. Training parameters are sent to the movement system (i.e., the underlying controller of the robot), which accurately controls motor output to drive the robot's linkage, achieving real-time, controlled physical movement and force interaction with the user.

[0179] Exemplarily, according to the assessment results, the regimen system generates a new training regimen:

[0180] Task type: upgrade from two-dimensional "straight line to reach" to three-dimensional "helical trajectory tracking".

[0181] Training intensity: increase baseline resistance from 5N to 8N.

[0182] Assistance / resistance mode: when the user's digital activation signal for biceps brachii is lower than 0.2, the robot provides N of assistance force; when the signal is higher than 0.2, the robot applies a constant resistance of 8N.

[0183] Subsequently, the rehabilitation robot starts executing this new plan, and the user trains in the new task and mechanical environment. In the execution process, all the previous steps are still running in the background at a high frequency, continuously collecting new data, continuously correcting the digital twin model, and continuously performing micro-assessment, thus forming a complete closed loop. If the system detects that the user has developed an undesirable compensation pattern (for example, through sEMG, it is found that the shoulder muscles are over-activated) under the new plan, the closed-loop system will immediately respond, possibly adjusting the assistance / resistance parameters within a few seconds to guide the user back to the correct movement pattern, and so on.

[0184] Embodiment Two

[0185] After obtaining the corrected model, the correction process may have overfitting, resulting in reduced prediction accuracy. Unlike Embodiment One, this embodiment detects the corrected model, specifically:

[0186] S341: Constructing a Virtual Exploration Task Set and a Benchmark Response Database

[0187] The system pre-constructs a standardized virtual exploration task set VPTS, which is independent of the user's daily rehabilitation training tasks. The task set stimulates various response characteristics of the user's neural-muscular-skeletal system in a standard manner. For each exploration task, the system maintains a benchmark database extracted from large-scale clinical data, which describes the expected physiological signal patterns (such as EEG ERD / ERS patterns, sEMG synergistic activation patterns, etc.) of healthy people or similar patient groups under specific exploration tasks.

[0188] S342: Periodically Perform Online Model Verification

[0189] The system initiates an online model verification process (this process does not involve physical interaction with real users and is completely performed in a computing environment) at a lower frequency (for example, after completing a complete rehabilitation training session, or after 30 minutes of training).

[0190] Model Snapshot: The system first obtains a complete "snapshot" of the fully corrected digital twin model (including cognitive-motor model, skeletal muscle control model, and musculoskeletal model) at the current time.

[0191] Virtual Execution of Exploration Tasks: The system drives this model snapshot to execute all virtual exploration tasks in the VPTS in a simulation environment one by one.

[0192] Generate Predicted Responses: For each exploration task, the model snapshot generates a complete set of predicted, multi-level physiological response data, including predicted brain electrical patterns, muscle activation patterns, and limb movement trajectories, etc.

[0193] S343: Calculate model evaluation metrics and generate validation reports.

[0194] The system compares the predicted responses generated above with the baseline response database corresponding to VPTS in multiple dimensions, and calculates a series of evaluation indicators for quantifying model quality:

[0195] Generalization performance score ( ): Used to measure the model’s performance on untrained exploration tasks, specifically by calculating the difference between the predicted response and the baseline response (e.g., Dynamic Time Warped (DTW) distance); the smaller the difference, the higher the score.

[0196] Predicted stability score ( ): Used to evaluate the consistency of a model's output under similar inputs. It is calculated by applying a small perturbation to the input of the exploration task and observing the degree of jitter in the model's output. The smaller the jitter, the higher the score.

[0197] Physiological explainability score ( This section examines whether the model's internal working mechanism conforms to known physiological principles. For example, when simulating a rapid elbow flexion task, does the output of the skeletal muscle control model clearly reflect the "triphasic activation pattern" where the agonist muscle (biceps brachii) is activated first, followed by braking of the antagonist muscle (triceps brachii)? The higher the conformity, the higher the score.

[0198] The system combines these scores to generate an online verification report on the current quality of the digital twin model.

[0199] S344: Adaptive adjustment strategy for model correction based on validation report

[0200] Based on the validation report generated by S343, the system dynamically adjusts the "meta-parameters" (especially the strength of the regularization term) used in subsequent model corrections. Regularization is a method used in machine learning (such as L1 / L2 regularization) to prevent model overfitting. Specifically, it involves adding a penalty term to the loss function to constrain the complexity of the model.

[0201] The adaptive adjustment logic is as follows:

[0202] If the validation report shows a generalization performance score Lower values ​​indicate that the model may have begun to overfit the current training task. In this case, the system will automatically increase the strength of the regularization term in the aforementioned loss function (e.g., increase the L2 regularization coefficient). This will allow subsequent model correction processes to focus more on maintaining the smoothness and simplicity of model parameters while reducing training errors, thereby improving its adaptability to new tasks.

[0203] If the validation report shows a generalization performance score High: This indicates that the model is in good condition and has good generalization ability. At this time, the system can appropriately reduce the regularization strength, allowing the model to fit the user's individual characteristics more precisely in order to pursue higher personalized accuracy.

[0204] For example, a user performs repetitive "horizontal trajectory tracking" training for a whole week.

[0205] Initially: Through continuous refinement, the model's prediction error for this task gradually decreases.

[0206] Model validation triggered: After one week of training, the system initiates online model validation.

[0207] Problem identified: When performing the "vertical anti-gravity lift" task in the virtual exploration task set, the muscle activation patterns predicted by the model snapshots were severely inconsistent with physiological benchmarks, demonstrating extremely poor generalization ability. In this case, the validation report gave a low score. Fraction.

[0208] Adaptive adjustment: The system identifies the risk of overfitting and automatically adjusts the L2 regularization coefficient in the skeletal muscle control model algorithm. Increased from 0.001 to 0.01.

[0209] Subsequent impact: In subsequent training, even on horizontal tasks, the model correction process will tend to learn muscle coordination strategies that are more in line with the basic principles of biomechanics due to stronger regularization constraints, rather than simply memorizing specific activation patterns for horizontal tasks, and so on.

[0210] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0211] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0212] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is arranged to execute the steps in any of the method embodiments when running.

[0213] In an example embodiment, the computer readable storage medium can include, but is not limited to, a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and various media capable of storing a computer program.

[0214] The embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is arranged to execute the computer program to perform the steps in any of the method embodiments.

[0215] In an example embodiment, the electronic device can further comprise a transmission device and an input and output device, wherein the transmission device is connected to the processor, and the input and output device is connected to the processor.

[0216] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0217] In the several embodiments provided in the present application, it should be understood that the disclosed device and method can be implemented by other ways. For example, the device embodiments described above are only illustrative, and for example, the division of the modules or units is only a logical function division, and there can be another division way in actual implementation, for example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0218] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place, or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0219] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0220] When the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on such understanding, the technical solutions of the embodiments of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, including a plurality of instructions to make a device (which can be a single-chip microcomputer, a chip, etc.) or a processor execute all or part of the steps of the method of each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0221] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multimodal neural-driven rehabilitation robot control system, characterized in that, include: The data acquisition module is used to acquire human-computer interaction task data and real-time user signals, wherein the user signals include electroencephalogram (EEG) signals, electromyogram (EMG) signals, and human-computer motion / force interaction data, and the human-computer interaction task data includes task type, task objective, and user operation feedback. The signal generation module is used to generate digital signals based on the user signals and a preset recognition system. The digital signals include digital motion signals representing the user's movement intentions generated based on the electroencephalogram (EEG) signals and a preset motion signal recognition system, and digital activation signals representing muscle activation states generated based on the electromyogram (EMG) signals and a preset activation signal recognition system. The module also generates simulation signals based on a preset model and the human-computer interaction task data. The simulation signals include simulated motion signals, simulated activation signals, and simulated motion / force data. The correction module is used to correct the corresponding preset models online based on the difference between the digital signal and the simulation signal, or based on the difference between the human-computer motion / force interaction data and the simulation motion / force data. The rehabilitation assessment module is used to generate rehabilitation assessment data based on the parameters of the corrected model and the acquired historical training data. The task generation module is used to generate task data and training parameters based on the rehabilitation assessment data. The task data is used for task information interaction between the rehabilitation robot and the user, and the training parameters are used for motion / force interaction between the rehabilitation robot and the user. The step of calculating the difference between the digital signal and the simulated signal includes: The digital motion signal is compared with the simulated motion signal output by the cognitive-motor model based on the human-computer interaction task data to obtain a first difference, and the cognitive-motor model is corrected online based on the first difference using a first machine learning algorithm, wherein the preset model includes the cognitive-motor model. The digital activation signal is compared with the simulated activation signal output by the skeletal muscle control model based on the digital motion signal to obtain a second difference. The skeletal muscle control model is then corrected online based on the second difference using a second machine learning algorithm. The preset model includes the skeletal muscle control model. Online correction of the preset model based on the difference between the human-computer motion / force interaction data and the simulated motion / force data includes: The real-time human-computer motion / force interaction data is compared with the simulated motion / force data output based on the musculoskeletal model according to the simulation activation signal to obtain a third difference. A third machine learning algorithm is then used to correct the musculoskeletal model online based on the third difference. The preset model includes the musculoskeletal model.

Images