Upper limb and lower limb collaborative rehabilitation training system based on multi-modal sensing

Abstract

The invention relates to the technical field of rehabilitation medicine, and discloses an upper limb and lower limb collaborative rehabilitation training system based on multi-modal sensing, comprising: a multi-modal sensor module for collecting motion data of upper limbs and lower limbs of a patient; the data processing module is used for receiving and processing the motion data transmitted by the multi-mode sensor module; the optimization control module is used for receiving the motion data and combining a rehabilitation target and coordination constraints; the feedback module is used for providing motion feedback of the patient according to the real-time training state and adjusting a motion control instruction in real time; and the display module is used for displaying the motion data, the training state and the real-time feedback information of the upper limbs and the lower limbs. Through data acquisition of the multi-modal sensor, dynamic adjustment of the optimization control module, real-time personalized feedback and immersive graphical display, accurate, efficient and personalized rehabilitation training is realized, and the training effect and the participation experience of a patient are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation medicine, and in particular to an upper limb and lower limb coordinated rehabilitation training system based on multimodal sensing. Background Art

[0002] With the aging of the population and the changes in modern lifestyles, sports rehabilitation has become increasingly important in the recovery of the elderly, postoperative patients, and patients with sports injuries. Rehabilitation training of the upper and lower limbs is not only a key step in restoring normal motor function, but also essential for improving the quality of life of patients. Traditional rehabilitation training equipment mostly relies on mechanized equipment and preset exercise programs, lacking intelligent feedback and real-time adjustment functions.

[0003] Existing rehabilitation training equipment has adopted motion data acquisition and motion control technology to a certain extent; these devices usually monitor the patient's motion status through sensors and guide the patient's training according to certain preset standards. Some equipment can provide certain feedback on the motion status and adjust the intensity or time of training to ensure the training effect. These technologies have improved the efficiency of rehabilitation training to a certain extent, enabling patients to obtain basic training guidance and feedback, and reducing sports injuries caused by improper training.

[0004] However, there are still many shortcomings in the existing technology. First, most traditional rehabilitation equipment relies on a single sensor for data collection, which leads to insufficient accuracy and comprehensiveness of the data and cannot fully reflect the patient's movement status. Secondly, the existing motion control solutions are usually inflexible and cannot dynamically adjust the training instructions according to the patient's real-time needs, resulting in the training effect cannot be maximized. The feedback mechanism also has lags and singleness, and patients cannot obtain timely and personalized adjustment suggestions during exercise. In addition, the training effect display method of existing equipment is relatively simple, lacking immersive and intuitive feedback, and it is difficult to mobilize patients' enthusiasm for training. These problems have led to the inability of traditional rehabilitation training equipment to fully meet individual needs, and the effect during training is not ideal. Summary of the invention

[0005] In view of the shortcomings of the existing technology, the present invention provides an upper and lower limb coordinated rehabilitation training system based on multimodal sensing, which aims to solve the problems in the existing technology of incomplete data collection, lack of personalized adjustment of motion control, lagging and single feedback mechanism, and poor intuitiveness in displaying training effects.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: an upper limb and lower limb coordinated rehabilitation training system based on multimodal sensing, comprising:

[0007] A multimodal sensor module for collecting motion data of the patient's upper and lower limbs;

[0008] A data processing module, used for receiving and processing the motion data transmitted by the multimodal sensor module;

[0009] The optimization control module generates coordinated motion control instructions for the upper and lower limbs by receiving motion data and combining rehabilitation goals and coordination constraints, wherein the coordinated motion control instructions are used to guide the patient's upper and lower limbs to perform training;

[0010] A feedback module is used to provide the patient with motion feedback according to the real-time training status and adjust the motion control instructions in real time;

[0011] The display module is used to display the motion data, training status and real-time feedback information of the upper limbs and lower limbs.

[0012] Preferably, the multimodal sensor module comprises:

[0013] Inertial measurement unit, used to obtain the motion trajectory data of the patient's upper and lower limbs;

[0014] Myoelectric sensors, used to obtain myoelectric activity data of upper and lower limbs;

[0015] Pressure sensors are used to monitor the distribution of pressure applied by the upper and lower limbs during exercise.

[0016] Preferably, the data processing module includes:

[0017] A Kalman filter for fusing data from the multimodal sensor;

[0018] The Kalman filter is used to estimate the patient's motion state in real time, including motion posture, acceleration and angle data, and use it for subsequent motion control and optimization calculations.

[0019] Preferably, the optimization control module includes:

[0020] An optimal control unit, for generating coordinated movement control instructions of the upper limbs and lower limbs according to input movement data and rehabilitation goals;

[0021] The coordination optimization unit ensures the best coordination and synchronization between the upper and lower limbs during exercise by optimizing the exercise intensity, speed and angle parameters.

[0022] Preferably, the feedback module includes:

[0023] Real-time monitoring unit, used to monitor the patient's movement status in real time and obtain movement data of upper and lower limbs;

[0024] The feedback generation unit is used to generate motion feedback information and provide training prompts to the patient through visual, sound or tactile feedback.

[0025] Preferably, the display module comprises:

[0026] Graphical display unit, used to display upper and lower limb motion data, training status and real-time feedback information;

[0027] The progress update unit updates the training progress, movement trajectory, rehabilitation progress and control command adjustment information in real time.

[0028] Preferably, the feedback generating unit comprises:

[0029] A motor assessment unit, used to assess the patient's upper and lower limb motor status;

[0030] The feedback instruction generating unit is used to generate and output appropriate training feedback, including visual, audio or tactile feedback.

[0031] Preferably, the graphical display unit comprises:

[0032] A motion data display unit, used to display the real-time motion data of the patient's upper and lower limbs;

[0033] The training progress display unit is used to display the patient's training status and progress information.

[0034] Preferably, the real-time monitoring unit comprises:

[0035] A motion state acquisition unit, used to acquire the motion data of the patient's upper and lower limbs in real time, including position, acceleration and angle data;

[0036] The data transmission unit is used to transmit the motion data to the feedback generation unit for subsequent processing.

[0037] Preferably, the progress updating unit comprises:

[0038] Training progress acquisition unit, used to obtain the patient's current exercise progress and rehabilitation status;

[0039] The status update unit updates the patient's training status and adjusts the training goals and parameters according to the training progress and motion data.

[0040] The present invention provides an upper and lower limb coordinated rehabilitation training system based on multimodal sensing.

[0041] Beneficial effects:

[0042] 1. The present invention adopts a technical solution of combining a multimodal sensor module with a data processing module, which can collect and process the patient's motion data in real time, including acceleration, angle, pressure and other information of the upper and lower limbs, and achieves accurate data fusion and motion state estimation technology effects. Compared with the solution of using a single sensor or a simple data processing method in the prior art, the present invention solves the problem of large error and lack of comprehensiveness of single sensor data, making the effect of rehabilitation training more accurate and reliable.

[0043] 2. The present invention adopts the technical solution of optimizing control module combined with rehabilitation goals and coordination constraints to generate accurate coordinated motion control instructions, achieving the technical effect of improving the coordination and synchronization of patient movements. Compared with the prior art solution that simply relies on fixed training programs, the present invention avoids training intensity that is not suitable for the actual situation of the patient by dynamically adjusting the motion control instructions, ensuring that each training link can be optimized according to the patient's real-time status.

[0044] 3. The present invention adopts a technical solution that generates personalized feedback information through real-time monitoring and data feedback using a feedback module, achieving the technical effect of effectively guiding and adjusting the patient's movement state. Compared with the prior art solution with a single feedback mechanism or delayed feedback, the present invention prompts the patient in real time through multiple feedback methods, solving the problem of delayed feedback and difficulty for patients to adjust training movements in real time in the traditional system, and improving the interactivity and effectiveness of rehabilitation training.

[0045] 4. The present invention adopts a technical solution combining display module with virtual reality or augmented reality to achieve a more intuitive and immersive display of training effects. Compared with the traditional two-dimensional data display in the prior art, the present invention allows patients to see the changes in the exercise process in real time through an immersive environment, which enhances the sense of participation and motivation in training, and greatly improves the training enthusiasm and rehabilitation effect of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the system construction of the present invention;

[0047] Figure 2 It is a framework diagram of a multimodal sensor module of the present invention;

[0048] Figure 3 It is a data processing module framework diagram of the present invention;

[0049] Figure 4 It is the framework diagram of the optimization control module of the present invention;

[0050] Figure 5 It is a framework diagram of the feedback module of the present invention;

[0051] Figure 6 It is a frame diagram of a display module of the present invention;

[0052] Figure 7 It is a framework diagram of the feedback generation unit of the present invention;

[0053] Figure 8 It is a framework diagram of a graphical display unit of the present invention;

[0054] Fig. 9 It is a framework diagram of the real-time monitoring unit of the present invention;

[0055] Fig.10 It is a framework diagram of the progress update unit of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Please see attached Figure 1 -Attached Fig.10 The embodiment of the present invention provides an upper limb and lower limb coordinated rehabilitation training system based on multimodal sensing, comprising:

[0058] A multimodal sensor module for collecting motion data of the patient's upper and lower limbs;

[0059] As a core component, the multimodal sensor module is responsible for collecting the movement data of the patient's upper and lower limbs. The design purpose of this module is to obtain comprehensive data related to the patient's movement through the cooperation of multiple high-precision sensors. Through the precise data collection of this module, the system can adjust the control strategy in real time according to different movement states and rehabilitation needs to ensure the efficiency and safety of exercise training.

[0060] The multimodal sensor module is used to collect the motion data of the patient's upper and lower limbs. The module includes multiple sensing units, which work together with different types of sensors to obtain dynamic information of the patient during exercise in real time. Specifically, the multimodal sensor module in this embodiment includes components such as an inertial measurement unit (IMU), an electromyography sensor (EMG), and a pressure sensor.

[0061] Inertial Measurement Unit (IMU): The inertial measurement unit uses a combination of multiple sensors such as accelerometers, gyroscopes and magnetometers to accurately measure the movement trajectory, acceleration and angle data of the patient's upper and lower limbs. Accelerometers are used to monitor the patient's acceleration, gyroscopes are used to measure the angular velocity of rotation, and magnetometers can provide direction information. These data can be used to describe the angle changes, movement trajectory and movement speed of the patient's joints. Specifically, the IMU can capture the dynamic changes of the patient's movement in real time, and convert these changes into digital signals to be transmitted to the subsequent data processing unit.

[0062] In one possible implementation, the inertial measurement unit collaborates with other sensors through data fusion technology to provide high-precision motion posture estimation. For example, the angular velocity signal ω(t) and acceleration signal a(t) obtained by the IMU sensor can be fused and processed by the following formula:

[0063] θ(t)=∫ω(t)dt

[0064] v(t)=∫a(t)dt

[0065] Among them, θ(t) represents the angle of the movement posture, and v(t) represents the speed of the patient's movement.

[0066] Electromyographic sensor (EMG): Electromyographic sensor is used to monitor the electromyographic activity of the patient's upper and lower limbs. By collecting electrical signals from muscle fibers, electromyographic sensors can reflect the activity intensity and fatigue level of muscles. Specifically, the amplitude and frequency of electromyographic signals are proportional to the state of muscle use. Electromyographic signals are collected through electrodes on the surface of the skin, and these signals can provide important basis for subsequent movement optimization control.

[0067] In some embodiments, the electromyographic signal can be pre-processed to obtain the frequency spectrum characteristics of muscle activity, and used to estimate the degree of muscle fatigue in real time so as to adjust the training intensity.

[0068] Pressure sensor: The pressure sensor is mainly used to monitor the pressure distribution of the patient's upper and lower limbs during exercise. The sensor is placed in the pressure-sensitive area of ​​the patient's contact surface to accurately reflect the patient's support during exercise. Through the data collected by the pressure sensor, the system can determine in real time whether the patient has an unbalanced load, thereby adjusting the exercise mode to prevent excessive load or unreasonable exercise posture and ensure the safety of training.

[0069] In this embodiment, the output signal of the pressure sensor can be analyzed by the following formula:

[0070]

[0071] Among them, Ptotal (t) is the total pressure measured by the sensor at a specific time point t, P i (t) is the pressure value of the i-th pressure sensor at time t, and n is the number of sensors.

[0072] The motion data collected by the multimodal sensor module in this embodiment will be transmitted to the subsequent data processing module for processing through the data transmission unit. During the data transmission process, the output signal of the sensor module will usually undergo preliminary signal conditioning and filtering to remove noise and ensure the accuracy of the data. The data transmission unit transmits this information to data processing units such as the Kalman filter through an efficient data channel, thereby realizing data fusion and state estimation. In subsequent data processing, the Kalman filter integrates the data of the inertial measurement unit (IMU), electromyography sensor, and pressure sensor to ensure accurate motion state estimation.

[0073] In some embodiments, to improve the accuracy and reliability of motion data, the sensor module can use more sensor types. For example, infrared sensors or optical sensors can be used to monitor the distance and position changes of patients during movement. In addition, this embodiment can also combine visual recognition technology to fuse the video data from the camera with the sensor data to further improve the comprehensiveness of motion data collection.

[0074] This embodiment can also be applied to different types of rehabilitation training equipment. Specifically, the inertial measurement unit and the pressure sensor can be embedded in the support surface or motion track of the rehabilitation training equipment to monitor the contact between the patient and the equipment in real time and adjust the equipment motion mode according to the real-time data.

[0075] A data processing module, used for receiving and processing the motion data transmitted by the multimodal sensor module;

[0076] The data processing module plays a key role in receiving, processing and analyzing motion data from the multimodal sensor module. The data collected by the multimodal sensor module through the inertial measurement unit (IMU), electromyography sensor (EMG) and pressure sensor can provide more accurate motion state estimation and control instructions after being processed by the data processing module, providing support for subsequent motion control and feedback generation. The data processing module ensures that the entire process from data collection to the generation of final control instructions can be completed accurately and efficiently, and denoises, filters and fuses the data, further improving the response speed and reliability of the system.

[0077] The data processing module is used to receive and process the motion data from the multimodal sensor module to ensure that the data collected by the sensor can be fully utilized. The module is mainly composed of multiple sub-units, including a Kalman filter, a data fusion unit, and a motion state estimation unit.

[0078] Kalman filter: Kalman filter is the core component of the data processing module. It is responsible for fusing data from various sensors, removing noise from sensor signals, and providing optimal motion state estimation.

[0079] Generally speaking, the original signals output by sensors often contain errors or noise. For example, the acceleration data of IMU may be affected by external disturbances, and the electromyography sensor signal may also be disturbed by environmental noise. In order to improve the estimation accuracy, the Kalman filter can optimally weight the multi-source data through recursive calculation, thereby providing a more accurate estimation under various interferences.

[0080] Specifically, the Kalman filter updates the state based on the state equation and observation equation of the system, using the observation data at the current moment and the estimation results at the previous moment. Assuming that the state quantity is x(t), the state transfer matrix of the system is A, the observation matrix is ​​H, the measurement noise covariance is R, and the process noise covariance is Q, the update formula of the Kalman filter is:

[0081] x k|k =x k|k-1 +K k (z k -Hx k|k-1 )

[0082] K k =P k|k-1 H T (HP k|k-1 H T +R) -1

[0083] P k|k =(IK k H)P k|k-1

[0084] Among them, K k is the Kalman gain, P k|k-1 is the prediction covariance matrix, P k|k is the updated covariance matrix, z k Through these formulas, the Kalman filter can optimize the fusion of IMU, EMG and pressure sensor data, reduce measurement errors, and provide more accurate motion state estimation.

[0085] Data fusion unit: The main task of the data fusion unit is to merge and process different data sources from multimodal sensors to further improve the accuracy and integrity of the data. In general, the acceleration and angular velocity data provided by the inertial measurement unit (IMU) can help estimate the posture of the movement, while the signal provided by the electromyography sensor reflects the activity state of the muscle, and the data of the pressure sensor can reflect the load distribution of the body during the movement.

[0086] In some embodiments, the data fusion unit uses a weighted average method or a weighted fusion based on a filtering algorithm to enhance the accuracy of data fusion by combining the characteristics of different sensors. For example, the position data and acceleration data provided by the IMU and pressure sensor can be fused by the following weighted combination formula:

[0087] X f =w1X IMU +w2X Pressure

[0088] Among them, X f is the fused data, X IMU and X Pressure The data are output by IMU and pressure sensor respectively, w1 and w2 are weighting coefficients, and w1+w2=1. By selecting the weighting coefficients reasonably, the fusion effect can be optimized according to the performance characteristics of each sensor.

[0089] Motion state estimation unit: The motion state estimation unit is responsible for calculating the patient's motion state in real time based on the fused data. Usually, this unit needs to estimate the patient's displacement, velocity, acceleration, posture angle and other key parameters based on the processed data. In the specific implementation, the motion state estimation unit can accurately estimate the patient's motion state by analyzing the real-time data and combining the aforementioned Kalman filter results. For example, the motion state estimation unit can derive the angle change and acceleration of each joint of the patient's upper and lower limbs through the output results of the Kalman filter, providing necessary data support for the motion control module.

[0090] Specifically, assuming that the velocity and position of the patient obtained by the Kalman filter are v(t) and p(t), the motion state can be further estimated by the following formula:

[0091] v(t)=∫a(t)dt

[0092] p(t)=∫v(t)dt

[0093] Among them, a(t) is acceleration, v(t) is velocity, and p(t) is displacement. The motion state estimation unit can make real-time motion state predictions based on these data, providing accurate basis for the decision-making of the motion control module.

[0094] In some embodiments, the data processing module may also include a motion pattern recognition function to further analyze the patient's motion type based on the processed data, such as gait, walking, running, etc. By identifying different motion patterns, the system can provide different types of feedback to further improve personalized rehabilitation effects.

[0095] In addition, this embodiment can also be applied to more complex rehabilitation training environments. For example, in combination with virtual reality technology, the data processing module can synchronize sensor data with scenes in virtual reality, adjust training tasks in the virtual environment in real time, and enable patients to perform more diverse rehabilitation training in an immersive environment.

[0096] The optimization control module generates coordinated motion control instructions for the upper and lower limbs by receiving motion data and combining rehabilitation goals and coordination constraints, wherein the coordinated motion control instructions are used to guide the patient's upper and lower limbs to perform training;

[0097] The optimization control module is a crucial component of the entire system. It is responsible for receiving motion data from the data processing module and generating coordinated motion control instructions for the upper and lower limbs according to the rehabilitation goals and coordination constraints. This module must not only make dynamic adjustments based on the patient's current state, but also ensure the coordination and synchronization between the upper and lower limbs to ensure the efficiency and safety of rehabilitation training.

[0098] In this embodiment, the optimization control module generates coordinated movement control instructions for the upper and lower limbs by receiving movement data and combining rehabilitation goals and coordination constraints. These control instructions can guide the patient to perform coordinated upper and lower limb training to help the patient restore optimal movement ability during the rehabilitation process.

[0099] The optimization control module mainly consists of two key units: the optimal control unit and the coordination optimization unit. These two units work together to ensure that the generated control instructions meet the rehabilitation goals and maintain coordination during exercise.

[0100] The main task of the optimal control unit is to generate coordinated movement control instructions for the upper and lower limbs based on the input motion data and the patient's rehabilitation goals.

[0101] In general, rehabilitation goals are set based on the individual conditions of the patient (such as motor ability, degree of recovery from illness, etc.). These goals may include: restoring normal gait, increasing the range of joint motion, improving muscle endurance, etc. The optimal control unit will generate specific motion instructions based on real-time motion data and these rehabilitation goals. For example, if the patient's goal is to restore gait, the optimal control unit will adjust the intensity and frequency of gait training by analyzing parameters such as gait cycle and stride length, and generate appropriate training instructions.

[0102] Specifically, the optimal control unit uses mathematical modeling and optimization algorithms to match the patient's motion state with the rehabilitation goal. Assuming that the patient's goal is to maximize the coordination of gait, the optimal control problem can be described by the following optimization objective function:

[0103]

[0104] Among them, u L (t) and u R (t) represent the motion control input of the left leg and the right leg respectively, x L (t) and x R (t represents the motion state of the left leg and the right leg respectively, α and β are weight coefficients, and T is the training period. This objective function achieves the coordinated motion of the upper and lower limbs by minimizing the motion difference between the left and right legs.

[0105] Coordination optimization unit: The main function of the coordination optimization unit is to ensure the best coordination and synchronization between the upper and lower limbs during the movement by optimizing the parameters such as movement intensity, speed, angle, etc. In some embodiments, the coordination optimization unit ensures the synchronization during the movement through constraints. For example, assuming that the patient is undergoing gait training, the coordination optimization unit ensures that the parameters such as the step frequency and stride length of the left and right legs in the gait cycle match. This process can be achieved through the following constraints:

[0106] u L (t) = u R (t),x L (t) = x R (t)

[0107] Among them, u L (t) and u R (t) is the motion control input for the left and right legs, x L (t) and x R (t) is the motion state of the left leg and the right leg. By optimizing the control algorithm, the coordination optimization unit adjusts the motion control instructions so that the left and right legs remain synchronized during the motion process, thereby maximizing the motion coordination.

[0108] In some embodiments, in order to better meet the individual needs of patients, the optimization control module can also adjust the motion control instructions in real time according to the patient's motion feedback. Specifically, the optimization control module can dynamically adjust the training plan in combination with the motion state data of the feedback module. For example, if the patient feels overly tired or uncoordinated during training, the optimization control module can reduce the burden on the patient by adjusting the exercise intensity or changing the exercise mode to ensure the safety and effectiveness of the training process.

[0109] In addition, the optimization control module can also adjust the training gait type and frequency in real time based on the patient's gait analysis results. For example, if the patient needs to restore a normal gait, the optimization control module can generate motion instructions that conform to the normal gait cycle and gradually guide the patient to restore a healthy gait pattern by adjusting the stride length, frequency and other parameters of each step.

[0110] In one possible implementation, the optimization control unit further improves the intelligence level of motion control by introducing artificial intelligence algorithms, such as deep learning or reinforcement learning. By training the neural network model, the optimization control unit can generate more accurate motion control instructions based on the patient's historical data and training feedback, thereby improving the rehabilitation effect.

[0111] A feedback module is used to provide the patient with motion feedback according to the real-time training status and adjust the motion control instructions in real time;

[0112] The feedback module is a key component of the entire system. Its main function is to provide effective motion feedback based on the patient's real-time training status and adjust the motion control instructions in real time based on this. Through this feedback mechanism, patients can get timely guidance during the training process to help them improve their sports performance and avoid incorrect movement patterns, so as to better achieve their rehabilitation goals. The implementation of the feedback module is closely linked to the aforementioned data processing module and optimization control module. It provides patients with personalized motion feedback by obtaining data from these modules and adjusts the motion control instructions according to the patient's real-time situation.

[0113] In this embodiment, the feedback module is used to provide the patient with motion feedback according to the real-time training status and adjust the motion control instructions in real time. Specifically, the feedback module includes a real-time monitoring unit and a feedback generation unit. These two units cooperate with each other to ensure that the feedback to the patient during the training process is timely and effective, and at the same time adjust the motion instructions according to the feedback results to optimize the training effect.

[0114] Real-time monitoring unit: The task of the real-time monitoring unit is to obtain the movement data of the upper and lower limbs by continuously tracking the patient's movement status, and transmit it to the feedback generation unit.

[0115] Generally, the real-time monitoring unit processes the motion data transmitted from the multimodal sensor module by connecting to the data processing module. The data includes but is not limited to the patient's motion speed, gait cycle, acceleration, joint angle and other information. This information reflects the patient's current motion performance and provides a basis for subsequent feedback generation.

[0116] As an option, the real-time monitoring unit can remove noise from sensor data and improve data accuracy through data fusion algorithms such as Kalman filtering. The processed real-time data will become an important basis for adjusting training intensity and control instructions.

[0117] Feedback generation unit: The feedback generation unit is responsible for generating personalized training feedback based on the patient's current exercise status and training goals.

[0118] Specifically, the feedback generation unit evaluates the patient's movement quality by analyzing the movement data. For example, if the patient's stride is too small or the step frequency is too fast during gait training, the feedback generation unit can generate relevant prompts in time to remind the patient to adjust his gait. The feedback content includes three forms: visual, auditory and tactile, ensuring that patients can get clear feedback in different training environments.

[0119] In some embodiments, the feedback generation unit automatically selects an appropriate feedback method based on the patient's athletic performance and training goals. For example, if the cadence during gait training is too high, the feedback generation unit can prompt the patient to make adjustments through voice prompts such as "The cadence is too fast, please adjust to a normal stride" or through vibration feedback.

[0120] The control output of the feedback generation unit can be described by the following formula:

[0121]

[0122] Where F(t) is the feedback signal generated at time t, f i (t) is the i-th feedback mode (e.g., vision, sound, touch, etc.), w i is the weight coefficient of the feedback method, and n is the total number of feedback methods. The setting of the weight coefficient can be adjusted based on the patient's preference or specific training goals.

[0123] Real-time adjustment of motion control instructions: The feedback module not only provides motion feedback, but also adjusts motion control instructions according to real-time feedback results. This function is directly connected with the work of the optimization control module. In a typical embodiment, if the patient becomes fatigued or uncoordinated during exercise, the feedback generation unit will detect this change and pass the feedback information to the optimization control module. The optimization control module adjusts parameters such as the intensity, speed or duration of the training based on the feedback information. For example, if the feedback generation unit detects that the patient's stride is less than the set standard, the optimization control module will appropriately increase the stride requirement to guide the patient to adjust the movement.

[0124] Specifically, after receiving the feedback information, the optimization control module will recalculate the training parameters and generate new motion control instructions to ensure the continuity and safety of the training. The update formula of the motion control instructions is as follows:

[0125] u L (t) = u L (t-1)+Δu L ,u R (t) = u R (t-1)+Δu R

[0126] Among them, u L (t) and u R (t) represent the motion control instructions for the left leg and the right leg at time ttt, respectively, Δu L and Δu R The feedback generation unit is used to adjust the control instructions. In this way, the motion control instructions can be matched with the patient's motion state in real time, ensuring efficient and safe training.

[0127] In some embodiments, in order to enhance the patient's training experience, the feedback module can also combine virtual reality (VR) or augmented reality (AR) technology to provide more intuitive and immersive training feedback. For example, the feedback generation unit can synchronously display the gait prompts in the virtual scene and the actual motion data based on the patient's real-time gait data, so that the patient can see the effect of his own exercise more intuitively.

[0128] In addition, the feedback module can also automatically adjust the feedback method based on the patient's long-term training data. For example, if a certain type of feedback (such as sound prompts) is more effective for the patient, the system can automatically increase the proportion of this type of feedback in subsequent training. In this way, the feedback module can continuously optimize its feedback strategy and improve the effect of rehabilitation training.

[0129] The display module is used to display the motion data, training status and real-time feedback information of the upper limbs and lower limbs.

[0130] The main function of the display module is to visualize the motion data, training status and real-time feedback information from the upper and lower limbs so that patients can intuitively understand their training progress and results. The design of the display module is closely connected with the previous data processing module, feedback module and optimization control module. Through real-time data display, patients can adjust their movements in time to improve the training effect. The display module not only provides a detailed display of motion data, but also helps patients better understand the training status through a visual and graphical interface to ensure the smooth progress of training.

[0131] The display module is used to display the upper and lower limbs' movement data, training status, and real-time feedback information. The display module consists of multiple functional units, including a graphical display unit and a progress update unit. These two units work together to ensure that patients can obtain relevant training data and feedback in real time and adjust their training plans as needed.

[0132] Graphical display unit: The main function of the graphic display unit is to display the patient's exercise data, training status and real-time feedback information intuitively and vividly.

[0133] Generally, the graphical display unit will display the motion data of the upper and lower limbs, such as motion trajectory, acceleration, angle change, etc., in the form of charts, curves or other graphical methods. These data are coordinated by the data processing module and the optimization control module, and after real-time calculation, they are presented to the patient in the form of dynamic graphics or static icons. For example, if the patient's stride is inconsistent during gait training, the graphical display unit can remind the patient to adjust his pace through changes in the stride curve. Specifically, the graphical display unit displays the motion data in the following ways:

[0134] S(t)=f(x L (t),x R (t),u L (t),u R (t))

[0135] Among them, S(t) is the motion data display graph at time t, x L (t) and x R (t) represents the motion state of the left leg and the right leg respectively, u L (t) and u R (t) is the corresponding movement control instruction. These display graphics can not only help patients quickly understand the movement status, but also provide intuitive feedback to promote patients' training consciousness.

[0136] Progress update unit: The progress update unit is responsible for updating the patient's training progress, movement trajectory, rehabilitation progress and control command adjustment information in real time.

[0137] In some embodiments, the progress update unit helps the patient understand his / her performance in training in real time by tracking and displaying various key indicators of training. For example, if the patient's training goal is to gradually increase the cadence, the progress update unit will display the cadence change curve in real time so that the patient can clearly see his / her progress and make necessary adjustments.

[0138] As an option, the progress update unit can also work closely with the feedback module. When the patient's exercise status or training progress deviates from the predetermined goal, the progress update unit will instantly display relevant information through color changes, annotation prompts, etc., to help patients adjust their training strategies in time.

[0139] During this process, the progress update unit updates the display content in the following ways:

[0140]

[0141] Among them, P(t) represents the progress of training, S(t) is the motion data displayed in real time, and the progress update unit will generate an updated graph or indicator of the training progress by accumulating data at each moment.

[0142] In some embodiments, the display module can be combined with virtual reality (VR) or augmented reality (AR) technology to enable patients to train in an immersive environment. Through VR / AR technology, the display module can superimpose motion data, training status, and feedback information in a virtual environment. Patients can not only see their own motion data, but also obtain more intuitive training feedback through guidance prompts in the virtual environment. For example, when patients are doing gait training, the virtual environment can dynamically display gait optimization suggestions to help patients better adjust their movements.

[0143] In addition, the display module can also be customized according to the individual needs of the patient. In some embodiments, the patient can adjust the displayed content and display mode through settings, such as choosing to present data in the form of charts, numbers or voice. For different patient groups, the display module provides a flexible display method to meet different needs.

[0144] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An upper and lower limb collaborative rehabilitation training system based on multimodal sensing, characterized in that: include: A multimodal sensor module for collecting motion data of the patient's upper and lower limbs; A data processing module, used for receiving and processing the motion data transmitted by the multimodal sensor module; The optimization control module generates coordinated motion control instructions for the upper and lower limbs by receiving motion data and combining rehabilitation goals and coordination constraints, wherein the coordinated motion control instructions are used to guide the patient's upper and lower limbs to perform training; A feedback module is used to provide the patient with motion feedback according to the real-time training status and adjust the motion control instructions in real time; The display module is used to display the motion data, training status and real-time feedback information of the upper limbs and lower limbs.

2. The upper and lower limbs coordinated rehabilitation training system based on multimodal sensing according to claim 1 is characterized in that: The multimodal sensor module comprises: Inertial measurement unit, used to obtain the motion trajectory data of the patient's upper and lower limbs; Myoelectric sensors, used to obtain myoelectric activity data of upper and lower limbs; Pressure sensors are used to monitor the distribution of pressure applied by the upper and lower limbs during exercise.

3. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 1 is characterized in that: The data processing module comprises: A Kalman filter for fusing data from the multimodal sensor; The Kalman filter is used to estimate the patient's motion state in real time, including motion posture, acceleration and angle data, and use it for subsequent motion control and optimization calculations.

4. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 1, characterized in that: The optimization control module includes: An optimal control unit, for generating coordinated movement control instructions of the upper limbs and lower limbs according to input movement data and rehabilitation goals; The coordination optimization unit ensures the best coordination and synchronization between the upper and lower limbs during exercise by optimizing the exercise intensity, speed and angle parameters.

5. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 1, characterized in that: The feedback module comprises: Real-time monitoring unit, used to monitor the patient's movement status in real time and obtain movement data of upper and lower limbs; The feedback generation unit is used to generate motion feedback information and provide training prompts to the patient through visual, sound or tactile feedback.

6. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 1, characterized in that: The display module comprises: Graphical display unit, used to display upper and lower limb motion data, training status and real-time feedback information; The progress update unit updates the training progress, movement trajectory, rehabilitation progress and control command adjustment information in real time.

7. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 5 is characterized in that: The feedback generating unit comprises: A motor assessment unit, used to assess the patient's upper and lower limb motor status; The feedback instruction generating unit is used to generate and output appropriate training feedback, including visual, audio or tactile feedback.

8. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 6, characterized in that: The graphical display unit comprises: A motion data display unit, used to display the real-time motion data of the patient's upper and lower limbs; The training progress display unit is used to display the patient's training status and progress information.

9. The upper and lower limb coordinated rehabilitation training system based on multimodal sensing according to claim 5, characterized in that: The real-time monitoring unit comprises: A motion state acquisition unit, used to acquire the motion data of the patient's upper and lower limbs in real time, including position, acceleration and angle data; The data transmission unit is used to transmit the motion data to the feedback generation unit for subsequent processing.

10. The upper and lower limbs coordinated rehabilitation training system based on multimodal sensing according to claim 6, characterized in that: The progress updating unit comprises: Training progress acquisition unit, used to obtain the patient's current exercise progress and rehabilitation status; The status update unit updates the patient's training status and adjusts the training goals and parameters according to the training progress and motion data.

Images