Peripheral nerve injury personalized rehabilitation system and method based on multi-modal large model

Abstract

The invention relates to the technical field of artificial intelligence assisted medical rehabilitation, in particular to a peripheral nerve injury personalized rehabilitation system and method based on a multi-modal large model, and the method comprises the steps: a feature fusion module employs space-time attention to fuse multi-modal time series data, and constructs an evaluation map; the personalized generation module is combined with historical data and reinforcement learning to generate a scheme containing virtual scene parameters; the interaction feedback module collects data through mixed reality and calculates action deviation; and the adaptive adjustment module adopts a meta-learning optimization model and a distributed iterative output scheme. According to the method, the cross-modal association precision of the motion features and the mechanical parameters is improved, dynamic matching of the training scene and the motion ability of the user is achieved, the virtual environment and the mechanical feedback threshold are optimized by dynamically adjusting the rehabilitation scheme parameters, the scheme optimization period is shortened based on an online iterative optimization mechanism, and the training efficiency is improved. The core defects of personalized adaptation lagging and low utilization efficiency of multi-modal data are overcome.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence-assisted medical rehabilitation technology, and in particular to a personalized rehabilitation system and method for peripheral nerve injury based on a multimodal large model. Background Art

[0002] Existing intelligent assisted training systems for adaptively optimizing user motor functions suffer from issues such as insufficient modeling of spatiotemporal correlations in multimodal data and limited dynamic adaptation capabilities for personalized solutions. Traditional methods often rely on static multimodal feature splicing or fixed weight fusion, which struggles to capture the dynamic correlations between time-series data such as motion posture and mechanical signals, leading to distorted cross-modal feature representation. Furthermore, solution generation modules often rely on preset rule libraries or single-shot inference models, failing to integrate real-time biomechanical feedback for online optimization of training parameters. This leads to flaws such as a poor match between the virtual scene and the user's actual motor abilities and inaccurate mechanical feedback threshold settings, impacting training effectiveness and user experience.

[0003] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a personalized rehabilitation system and method for peripheral nerve injury based on a multimodal large model.

[0005] In a first aspect, the present invention provides a personalized rehabilitation system for peripheral nerve injury based on a multimodal large model. The technical solution of the system is as follows:

[0006] The personalized rehabilitation system for peripheral nerve injury based on the multimodal large model includes: a feature fusion module, a personalized generation module, an interactive feedback module and an adaptive adjustment module;

[0007] The feature fusion module is used to: dynamically weighted fuse the acquired multimodal time series data through a spatiotemporal attention mechanism, extract cross-modal correlation feature vectors, and construct a motor function evaluation map based on an improved graph convolutional network;

[0008] The personalized generation module is used to: input the motor function assessment map into a pre-trained multimodal large model, combine the user's historical rehabilitation data and environmental parameters, and generate a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning, and mechanical feedback thresholds through a reinforcement learning strategy;

[0009] The interactive feedback module is used to: render a virtual training scene of the personalized rehabilitation program in real time through a mixed reality device, synchronously collect biomechanical response data of the user in the virtual training scene, and calculate the user action execution deviation coefficient based on a dynamic time warping algorithm;

[0010] The adaptive adjustment module is used to: according to the action execution degree deviation coefficient, use the meta-learning framework to incrementally optimize the weight parameters of the multimodal large model, generate an incrementally optimized rehabilitation plan, and iterate the model parameters of the multimodal large model online through distributed computing nodes until the target rehabilitation plan is output.

[0011] The beneficial effects of the personalized rehabilitation system for peripheral nerve injury based on a multimodal large model of the present invention are as follows:

[0012] The system of the present invention improves the cross-modal correlation accuracy between motion characteristics and mechanical parameters, realizes dynamic matching of training scenarios with user motor abilities, optimizes the virtual environment and mechanical feedback thresholds by dynamically adjusting rehabilitation program parameters, and shortens the program optimization cycle based on an online iterative optimization mechanism, solving the core defects of personalized adaptation lag and low efficiency in multimodal data utilization.

[0013] On the basis of the above solution, the personalized rehabilitation system for peripheral nerve injury based on a multimodal large model of the present invention can be further improved as follows.

[0014] In an optional manner, the device further includes a data acquisition module; the data acquisition module is configured to:

[0015] Acquire motion posture data through a multi-axis inertial sensor array; acquire plantar pressure data through a three-dimensional pressure sensing device; and acquire physiological index data through a bioelectric signal acquisition unit.

[0016] The motion posture data, the plantar pressure data and the physiological index data are time synchronized and modally aligned to generate the multimodal time series data.

[0017] In the above optional method, the multimodal data synchronization alignment mechanism of the multi-axis inertial sensor, the three-dimensional pressure sensing device and the bioelectric signal acquisition unit is used to effectively eliminate the timestamp offset problem of different flow sensing devices, improve the cross-modal data fusion efficiency of motion posture, plantar pressure and physiological indicators, avoid the feature correlation error caused by data asynchrony, and provide a high-precision time series data foundation for subsequent analysis.

[0018] In an optional manner, the feature fusion module is specifically used to:

[0019] Inputting the motion posture data and the plantar pressure data into the temporal convolution branch and the spatial graph attention branch of the spatiotemporal attention mechanism respectively to generate a temporal dimension feature map and a spatial correlation weight matrix;

[0020] Performing dynamic channel weighted fusion on the time dimension feature map according to the spatial correlation weight matrix to generate a cross-modal correlation feature vector;

[0021] The cross-modal correlation feature vector is input into the graph convolutional network, and the motor function assessment map containing quantitative indicators of the degree of motor function impairment is output; wherein the adjacency matrix of the graph convolutional network is dynamically updated according to the physiological constraint relationship of the user's joint motion chain.

[0022] In the above optional methods, the branch processing strategy of the spatiotemporal attention mechanism combined with dynamic channel weighted fusion solves the problem of insufficient modeling of the spatial correlation between the spatiotemporal characteristics of motion posture and plantar pressure in traditional methods, improves the dynamic adjacency matrix update mechanism of the graph convolutional network, and makes the motion function evaluation map more in line with the changes in the physiological characteristics of the user's joint motion chain, thereby improving the accuracy of quantitative indicator generation.

[0023] In an optional manner, the personalized generation module is specifically configured to:

[0024] Inputting the motor function assessment atlas into the encoder branch of the multimodal large model, fusing the motor pattern feature vector in the user's historical rehabilitation data and the spatial layout feature vector in the environmental parameters through a multi-head cross-attention mechanism to generate a joint embedding representation;

[0025] Based on the joint embedding representation, a double-delayed deep deterministic policy gradient algorithm is used to search for a strategy that maximizes the action execution reward function in the virtual reality scene parameter space to generate the personalized rehabilitation plan; wherein, the action execution reward function dynamically constrains the mechanical feedback threshold according to the muscle activation degree coefficient in the physiological indicator data.

[0026] In the above optional method, the multimodal feature fusion based on the multi-head cross-attention mechanism and the dual-delay deep deterministic policy gradient algorithm realizes the coordinated optimization of the user's historical motion pattern, environmental space parameters and real-time mechanical feedback threshold. The mechanical parameters are dynamically constrained by the muscle activation degree coefficient, avoiding training errors caused by the mismatch between virtual scene parameters and the user's actual motion ability.

[0027] In an optional manner, the interactive feedback module is specifically configured to:

[0028] The eye tracking unit and force feedback gloves of the mixed reality device are used to collect the user's visual gaze hotspot data and joint torque response data in the virtual training scene to form the biomechanical response data;

[0029] Using a dynamic time warping algorithm, the joint torque response data is aligned frame by frame with the expected mechanical trajectory in the personalized rehabilitation program, the cumulative integral value of the joint torque deviation is calculated, and the cumulative integral value is used as the initial action execution degree deviation coefficient;

[0030] According to the heat weight of the visual gaze hotspot data, the initial action execution degree deviation coefficient is spatially weighted to generate the action execution degree deviation coefficient.

[0031] In the above optional method, by integrating the multi-dimensional biomechanical data collection of eye tracking and force feedback gloves, and combining the frame-level mechanical trajectory alignment and spatial attention weighting mechanism of the dynamic time warping algorithm, the error amplification problem caused by the uneven distribution of visual attention points in traditional motion deviation assessment is solved, and the spatial perception accuracy of the motion execution deviation coefficient is improved.

[0032] In an optional manner, the adaptive adjustment module is specifically configured to:

[0033] Comparing the action execution degree deviation coefficient with a preset convergence threshold, and if the convergence threshold is not reached, using a fast adaptation algorithm based on model-agnostic meta-learning in the meta-learning framework to perform a sub-network mask update on the encoder weight matrix of the multimodal large model to obtain a target sub-network mask;

[0034] Performing asynchronous gradient aggregation on the target subnetwork mask and the decoder weights of the multimodal large model through a parameter server in the distributed computing node to generate incrementally optimized target model parameters;

[0035] Updating the multimodal large model based on the target model parameters to generate an incrementally optimized rehabilitation plan;

[0036] According to the decreasing rate of the action execution degree deviation coefficient in the historical iteration rounds, the value of the convergence threshold is dynamically adjusted, and the target model parameters are synchronized to the decoder weight storage unit of the multimodal large model until the target rehabilitation plan is output.

[0037] Among the above optional methods, the sub-network mask update strategy based on model-agnostic meta-learning and distributed asynchronous gradient aggregation technology realizes incremental local optimization of multimodal large model parameters, avoiding the waste of computing resources caused by global parameter retraining. At the same time, through the dynamic adjustment mechanism of the convergence threshold, it can accelerate the iterative convergence speed of the model, thereby improving the optimization efficiency of the rehabilitation plan.

[0038] In an optional manner, the system further includes a constraint optimization module; the constraint optimization module is configured to:

[0039] Inputting the personalized rehabilitation plan into a pre-trained generative adversarial network discriminator to determine a feasibility score;

[0040] If the feasibility score is lower than a preset safety threshold, joint range of motion inequality constraints are applied to the parameters of the motion trajectory planning to regenerate the personalized rehabilitation plan that meets biomechanical safety conditions.

[0041] In the above optional method, the feasibility scoring mechanism of the generative adversarial network discriminator and the joint range of motion inequality constraints are used to ensure that the motion trajectory planning complies with the biomechanical safety criteria. The intelligent constraint parameter regeneration strategy is used to effectively avoid the action risks caused by exceeding the mechanical feedback threshold in the virtual training scenario.

[0042] In a second aspect, the present invention provides a personalized rehabilitation method for peripheral nerve injury based on a multimodal large model. The technical solution of the method is as follows:

[0043] The acquired multimodal time series data is dynamically weighted and fused through a spatiotemporal attention mechanism to extract cross-modal correlation feature vectors, and a motor function evaluation map is constructed based on an improved graph convolutional network.

[0044] The motor function assessment map is input into a pre-trained multimodal large model, combined with the user's historical rehabilitation data and environmental parameters, and a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning and mechanical feedback thresholds is generated through a reinforcement learning strategy;

[0045] Rendering a virtual training scene of the personalized rehabilitation program in real time through a mixed reality device, synchronously collecting biomechanical response data of the user in the virtual training scene, and calculating the user's action execution deviation coefficient based on a dynamic time warping algorithm;

[0046] According to the action execution degree deviation coefficient, a meta-learning framework is used to incrementally optimize the weight parameters of the multimodal large model to generate an incrementally optimized rehabilitation plan. The model parameters of the multimodal large model are then iterated online through distributed computing nodes until the target rehabilitation plan is output.

[0047] The beneficial effects of the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of the present invention are as follows:

[0048] The method of the present invention improves the cross-modal correlation accuracy of motion characteristics and mechanical parameters, realizes dynamic matching of training scenarios with user motor abilities, optimizes the virtual environment and mechanical feedback thresholds by dynamically adjusting rehabilitation program parameters, and shortens the program optimization cycle based on an online iterative optimization mechanism, solving the core defects of personalized adaptation lag and low efficiency of multimodal data utilization.

[0049] On the basis of the above scheme, the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of the present invention can be further improved as follows.

[0050] In an optional manner, the method further includes:

[0051] Acquire motion posture data through a multi-axis inertial sensor array; acquire plantar pressure data through a three-dimensional pressure sensing device; and acquire physiological index data through a bioelectric signal acquisition unit.

[0052] The motion posture data, the plantar pressure data and the physiological index data are time synchronized and modally aligned to generate the multimodal time series data.

[0053] In the above optional method, the multimodal data synchronization alignment mechanism of the multi-axis inertial sensor, the three-dimensional pressure sensing device and the bioelectric signal acquisition unit is used to effectively eliminate the timestamp offset problem of different flow sensing devices, improve the cross-modal data fusion efficiency of motion posture, plantar pressure and physiological indicators, avoid the feature correlation error caused by data asynchrony, and provide a high-precision time series data foundation for subsequent analysis.

[0054] In an optional approach, the steps of dynamically weighting and fusing the acquired multimodal time series data through a spatiotemporal attention mechanism, extracting cross-modal correlation feature vectors, and constructing a motor function evaluation map based on an improved graph convolutional network include:

[0055] Inputting the motion posture data and the plantar pressure data into the temporal convolution branch and the spatial graph attention branch of the spatiotemporal attention mechanism respectively to generate a temporal dimension feature map and a spatial correlation weight matrix;

[0056] Performing dynamic channel weighted fusion on the time dimension feature map according to the spatial correlation weight matrix to generate a cross-modal correlation feature vector;

[0057] The cross-modal correlation feature vector is input into the graph convolutional network, and the motor function assessment map containing quantitative indicators of the degree of motor function impairment is output; wherein the adjacency matrix of the graph convolutional network is dynamically updated according to the physiological constraint relationship of the user's joint motion chain.

[0058] In the above optional methods, the branch processing strategy of the spatiotemporal attention mechanism combined with dynamic channel weighted fusion solves the problem of insufficient modeling of the spatial correlation between the spatiotemporal characteristics of motion posture and plantar pressure in traditional methods, improves the dynamic adjacency matrix update mechanism of the graph convolutional network, and makes the motion function evaluation map more in line with the changes in the physiological characteristics of the user's joint motion chain, thereby improving the accuracy of quantitative indicator generation.

[0059] In an optional method, the motor function assessment map is input into a pre-trained multimodal large model, and combined with the user's historical rehabilitation data and environmental parameters, a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning, and mechanical feedback thresholds is generated through a reinforcement learning strategy, including the following steps:

[0060] Inputting the motor function assessment atlas into the encoder branch of the multimodal large model, fusing the motor pattern feature vector in the user's historical rehabilitation data and the spatial layout feature vector in the environmental parameters through a multi-head cross-attention mechanism to generate a joint embedding representation;

[0061] Based on the joint embedding representation, a double-delayed deep deterministic policy gradient algorithm is used to search for a strategy that maximizes the action execution reward function in the virtual reality scene parameter space to generate the personalized rehabilitation plan; wherein, the action execution reward function dynamically constrains the mechanical feedback threshold according to the muscle activation degree coefficient in the physiological indicator data.

[0062] In the above optional method, the multimodal feature fusion based on the multi-head cross-attention mechanism and the dual-delay deep deterministic policy gradient algorithm realizes the coordinated optimization of the user's historical motion pattern, environmental space parameters and real-time mechanical feedback threshold. The mechanical parameters are dynamically constrained by the muscle activation degree coefficient, avoiding training errors caused by the mismatch between virtual scene parameters and the user's actual motion ability.

[0063] In an optional embodiment, the steps of rendering a virtual training scene of the personalized rehabilitation program in real time using a mixed reality device, synchronously collecting biomechanical response data of the user in the virtual training scene, and calculating the user's action execution deviation coefficient based on a dynamic time warping algorithm include:

[0064] The eye tracking unit and force feedback gloves of the mixed reality device are used to collect the user's visual gaze hotspot data and joint torque response data in the virtual training scene to form the biomechanical response data;

[0065] Using a dynamic time warping algorithm, the joint torque response data is aligned frame by frame with the expected mechanical trajectory in the personalized rehabilitation program, the cumulative integral value of the joint torque deviation is calculated, and the cumulative integral value is used as the initial action execution degree deviation coefficient;

[0066] According to the heat weight of the visual gaze hotspot data, the initial action execution degree deviation coefficient is spatially weighted to generate the action execution degree deviation coefficient.

[0067] In the above optional method, by integrating the multi-dimensional biomechanical data collection of eye tracking and force feedback gloves, and combining the frame-level mechanical trajectory alignment and spatial attention weighting mechanism of the dynamic time warping algorithm, the error amplification problem caused by the uneven distribution of visual attention points in traditional motion deviation assessment is solved, and the spatial perception accuracy of the motion execution deviation coefficient is improved.

[0068] In an optional manner, based on the action execution degree deviation coefficient, a meta-learning framework is used to incrementally optimize the weight parameters of the multimodal large model to generate an incrementally optimized rehabilitation plan, and the model parameters of the multimodal large model are iterated online through distributed computing nodes until a target rehabilitation plan is output, including the following steps:

[0069] Comparing the action execution degree deviation coefficient with a preset convergence threshold, and if the convergence threshold is not reached, using a fast adaptation algorithm based on model-agnostic meta-learning in the meta-learning framework to perform a sub-network mask update on the encoder weight matrix of the multimodal large model to obtain a target sub-network mask;

[0070] Performing asynchronous gradient aggregation on the target subnetwork mask and the decoder weights of the multimodal large model through a parameter server in the distributed computing node to generate incrementally optimized target model parameters;

[0071] Updating the multimodal large model based on the target model parameters to generate an incrementally optimized rehabilitation plan;

[0072] According to the decreasing rate of the action execution degree deviation coefficient in the historical iteration rounds, the value of the convergence threshold is dynamically adjusted, and the target model parameters are synchronized to the decoder weight storage unit of the multimodal large model until the target rehabilitation plan is output.

[0073] Among the above optional methods, the sub-network mask update strategy based on model-agnostic meta-learning and distributed asynchronous gradient aggregation technology realizes incremental local optimization of multimodal large model parameters, avoiding the waste of computing resources caused by global parameter retraining. At the same time, through the dynamic adjustment mechanism of the convergence threshold, it can accelerate the iterative convergence speed of the model, thereby improving the optimization efficiency of the rehabilitation plan.

[0074] In an optional manner, the method further includes:

[0075] Inputting the personalized rehabilitation plan into a pre-trained generative adversarial network discriminator to determine a feasibility score;

[0076] If the feasibility score is lower than a preset safety threshold, joint range of motion inequality constraints are applied to the parameters of the motion trajectory planning to regenerate the personalized rehabilitation plan that meets biomechanical safety conditions.

[0077] In the above optional method, the feasibility scoring mechanism of the generative adversarial network discriminator and the joint range of motion inequality constraints are used to ensure that the motion trajectory planning complies with the biomechanical safety criteria. The intelligent constraint parameter regeneration strategy is used to effectively avoid the action risks caused by exceeding the mechanical feedback threshold in the virtual training scenario.

[0078] In a third aspect, the technical solution of an electronic device of the present invention is as follows:

[0079] The system comprises a memory, a processor and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of the present invention are implemented.

[0080] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0081] The computer-readable storage medium stores instructions. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of the present invention.

[0082] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0084] Figure 1 Schematic diagram of the structure of an embodiment of a personalized rehabilitation system for peripheral nerve injury based on a multimodal large model of the present invention;

[0085] Figure 2 Schematic diagram of a process of an embodiment of a personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of the present invention;

[0086] Figure 3 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0087] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0088] Figure 1 FIG. 1 shows a schematic structural diagram of an embodiment of a peripheral nerve injury personalized rehabilitation system 200 based on a multimodal large model provided by the present invention, as shown in FIG. Figure 1 As shown, the system 200 includes: a feature fusion module 210, a personalized generation module 220, an interactive feedback module 230 and an adaptive adjustment module 240;

[0089] The feature fusion module 210 is used to: dynamically weighted fuse the acquired multimodal time series data through the spatiotemporal attention mechanism, extract cross-modal correlation feature vectors, and construct a motor function evaluation map based on the improved graph convolutional network;

[0090] The personalized generation module 220 is used to: input the motor function assessment map into the pre-trained multimodal large model, combine the user's historical rehabilitation data and environmental parameters, and generate a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning, and mechanical feedback thresholds through a reinforcement learning strategy;

[0091] The interactive feedback module 230 is used to render a virtual training scene of the personalized rehabilitation program in real time through a mixed reality device, simultaneously collect the user's biomechanical response data in the virtual training scene, and calculate the user's action execution deviation coefficient based on a dynamic time warping algorithm;

[0092] The adaptive adjustment module 240 is used to: according to the action execution degree deviation coefficient, use the meta-learning framework to incrementally optimize the weight parameters of the multimodal large model, generate an incrementally optimized rehabilitation plan, and iterate the model parameters of the multimodal large model online through distributed computing nodes until the target rehabilitation plan is output.

[0093] Specifically, a motion function evaluation map is constructed through the dynamic weighted fusion of spatiotemporal attention of multimodal time series data and the improved graph convolutional network. Combined with user historical data and environmental parameters, a reinforcement learning strategy is used to generate virtual scene parameters, motion trajectories and mechanical feedback thresholds. Based on the mixed reality device, the training scene is rendered in real time and the user's biomechanical response data is collected. The action execution deviation coefficient is calculated through the dynamic time warping algorithm. The meta-learning framework is used to incrementally optimize the model parameters, and the distributed computing nodes are combined to realize the online iterative update of the multimodal large model, forming a closed-loop technical process of multimodal feature fusion, personalized solution generation, real-time deviation feedback and adaptive parameter adjustment, and finally outputting an intelligent optimization solution that meets the user's motion characteristics and environmental constraints.

[0094] The technical solution of this embodiment improves the cross-modal correlation accuracy between motion characteristics and mechanical parameters, realizes dynamic matching of training scenarios and user motion capabilities, optimizes the virtual environment and mechanical feedback thresholds by dynamically adjusting rehabilitation program parameters, and shortens the program optimization cycle based on the online iterative optimization mechanism, solving the core defects of personalized adaptation lag and low efficiency of multimodal data utilization.

[0095] In an optional manner, the device further includes a data acquisition module; the data acquisition module is configured to:

[0096] Acquire motion posture data through a multi-axis inertial sensor array; acquire plantar pressure data through a three-dimensional pressure sensing device; and acquire physiological index data through a bioelectric signal acquisition unit.

[0097] The motion posture data, plantar pressure data and physiological index data are time synchronized and modally aligned to generate multimodal time series data.

[0098] In this embodiment, the multi-axis inertial sensor array refers to a distributed network composed of multiple 6-axis or 9-axis MEMS (micro-electromechanical systems) inertial measurement units (IMUs). Each IMU contains a 3-axis accelerometer and a 3-axis gyroscope, and can also integrate a magnetometer. Among them, the accelerometer is used to measure linear acceleration; the gyroscope is used to measure angular velocity; and the magnetometer is used to measure the direction of the magnetic field. The multi-axis inertial sensor array is used to collaboratively measure the spatial posture changes of the human joint motion chain through multiple nodes, such as wrist flexion and extension angles, knee joint rotation angular velocity, etc., and output motion posture data.

[0099] The three-dimensional pressure sensing device is a matrix sensor composed of micro pressure-sensitive units (such as silicon strain gauges or conductive rubber) distributed on a flexible substrate based on the principle of piezoresistive or capacitive sensing. It is used to detect the pressure distribution and dynamic changes in each area. For example, an insole sensor with 1024 piezoresistive units is used to capture the pressure center transfer trajectory during the gait cycle at a sampling rate of 200Hz, and the force vector of the foot is inverted by the finite element model to output the plantar pressure data. The core parameters of the three-dimensional pressure sensing device include the range (0-50N / cm 2 ), spatial resolution (5mm×5mm) and dynamic response time (<10ms).

[0100] The bioelectric signal acquisition unit consists of a surface electromyography (EMG) sensor and electrocardiogram (ECG) electrodes. It extracts microvolt-level physiological electrical signals through a differential amplifier circuit and bandpass filtering. For example, an Ag / AgCl electrode is attached to the biceps brachii to capture the action potential waveform during muscle activation. A 50Hz notch circuit eliminates power frequency interference and outputs the muscle activation coefficient, thereby generating physiological indicator data. Technical parameters for the bioelectric signal acquisition unit include input impedance (≥1GΩ), common-mode rejection ratio (≥120dB), and sampling rate (2kHz).

[0101] Specifically, for time synchronization and modal alignment, the sampling clocks of the multi-axis inertial sensor array and the three-dimensional pressure sensing device are first synchronized via hardware interrupt trigger signals. Secondly, the high-sampling-rate data of the bioelectric signal acquisition unit is downsampled using linear interpolation to align the physiological indicator data timestamps with the motion posture data. Subsequently, a dynamic time warping algorithm is used to match the characteristic event points of the different modal data. Finally, a multimodal time series data matrix with strictly aligned time axes is generated, i.e., multimodal time series data. Each row of the matrix contains the synchronized joint angles, plantar pressure distribution, and muscle activation coefficients at the corresponding moment.

[0102] In an optional manner, the feature fusion module 210 is specifically configured to:

[0103] The motion posture data and plantar pressure data are respectively input into the temporal convolution branch and spatial graph attention branch of the spatiotemporal attention mechanism to generate a temporal dimension feature map and a spatial correlation weight matrix;

[0104] Perform dynamic channel weighted fusion on the time dimension feature map according to the spatial correlation weight matrix to generate a cross-modal correlation feature vector;

[0105] The cross-modal correlation feature vector is input into the graph convolutional network, and a motor function assessment map containing quantitative indicators of the degree of motor function impairment is output; the adjacency matrix of the graph convolutional network is dynamically updated according to the physiological constraint relationship of the user's joint motion chain.

[0106] In this embodiment, the time convolution branch refers to a temporal feature extraction network built based on dilated convolution. The core is to capture the motion patterns of different time spans in motion posture data through a convolution kernel with an adjustable dilation rate. For example, in gait analysis, setting a convolution layer with an expansion rate of 3 can cover the trend of motion acceleration changes from the previous 1 second to the current moment, and extract the periodic characteristics of the knee joint angular velocity. This branch outputs a time dimension feature map with a dimension of (number of time steps × number of feature channels), which is used to characterize the time domain evolution law of the motion sequence.

[0107] The spatial graph attention branch refers to a spatial correlation modeling module built on a graph attention network. It generates a spatial correlation weight matrix by calculating the dynamic correlation weights between user joint nodes. For example, human joints are modeled as graph nodes (e.g., shoulder, elbow, and wrist are adjacent nodes). A multi-head attention mechanism is used to calculate the correlation strength between the wrist and shoulder joints during flexion and extension. The output is a weight matrix of (number of nodes × number of nodes) dimensions, reflecting the spatial coupling relationship of the kinematic chain.

[0108] Dynamic channel weighted fusion is a feature fusion method based on the channel attention mechanism. It dynamically adjusts the channel weights of the temporal feature map according to the spatial correlation weight matrix. Furthermore, for each feature channel output by the temporal convolution branch, the average correlation between each feature channel and the corresponding node in the spatial graph attention branch is calculated, and a normalized weight ranging from 0 to 1 is generated using a sigmoid function. For example, when the spatial correlation weight between the knee angular velocity channel and the hip node is 0.8, this channel is enhanced to 1.8 times its original value during fusion, suppressing the contribution of irrelevant noise channels.

[0109] Dynamic updating of the adjacency matrix refers to the real-time adjustment of the graph convolutional network's adjacency matrix based on the physiological constraints of the user's joint kinematic chain (such as joint range of motion and ligament stretch limits). For example, during elbow flexion, if the current angle exceeds a preset safety threshold (such as 150°), a gating mechanism will attenuate the connection weight between that node and its adjacent nodes by 50%, thereby limiting the spread of abnormal movement patterns during feature propagation.

[0110] Specifically, the motion posture data and plantar pressure data are input into the temporal convolution branch and spatial graph attention branch of the spatiotemporal attention mechanism respectively. Among them, the temporal convolution branch uses a three-layer dilated convolution network with dilation rates of 2, 4, and 8 to perform multi-scale feature extraction on the temporal dimension of the motion posture data, and generates a time dimension feature map containing the acceleration gradient changes within the time window. The spatial graph attention branch calculates the dynamic correlation strength between each node through a multi-head attention mechanism based on the graph structure composed of human joint nodes, and generates a spatial correlation weight matrix that represents the collaborative relationship of the joint motion chain.

[0111] Dynamic channel weighted fusion of the time dimension feature map is performed according to the spatial correlation weight matrix. Specifically, the average correlation between each feature channel of the time dimension feature map and the corresponding joint node is input into the trainable scaling factor, the channel weight coefficient is generated through the normalization function, and the channel feature values ​​of the time dimension feature map are weighted and summed according to the channel weight coefficient to generate a cross-modal correlation feature vector; among which, the trainable scaling factor is initialized according to the statistical distribution of the channel and joint correlation in the historical training data, and the normalization function uses the Sigmoid function to constrain the weight coefficient to the range of 0 to 1.

[0112] The cross-modal correlation feature vector is input into the graph convolutional network, where the adjacency matrix of the graph convolutional network is dynamically updated according to the physiological constraints of the user's joint motion chain. Specifically, the connection strength adjustment coefficient between each node is calculated based on the real-time joint angle data. When the joint angle exceeds the preset safety threshold, the connection weight of the corresponding node is reduced through the gating mechanism. The adjusted adjacency matrix is ​​input into the Chebyshev polynomial approximation filter of the graph convolution layer, and the motor function assessment map containing the quantitative parameters of the motor function is output. The connection strength adjustment coefficient is calculated by the following formula:

[0113]

[0114] Among them, A ij is the updated adjacency weight between node i and node j; σ is the Sigmoid function; β is the trainable gating parameter; θ i is the current joint angle; θ safe θ is the preset joint safety angle; limit The maximum range of motion of the joint.

[0115] In an optional manner, the personalized generation module 220 is specifically configured to:

[0116] The motor function assessment map is input into the encoder branch of the multimodal large model. The motor pattern feature vector in the user's historical rehabilitation data and the spatial layout feature vector in the environmental parameters are fused through a multi-head cross-attention mechanism to generate a joint embedding representation.

[0117] Based on the joint embedding representation, a double-delayed deep deterministic policy gradient algorithm is used to search for a strategy that maximizes the action execution reward function in the parameter space of the virtual reality scene to generate a personalized rehabilitation plan; among them, the action execution reward function dynamically constrains the mechanical feedback threshold according to the muscle activation degree coefficient in the physiological indicator data.

[0118] In this embodiment, the motor function assessment atlas is constructed using a graph convolutional network (GCN) to collect data such as joint angles, muscle activation timing, and center of gravity shift trajectory using multi-dimensional biomechanical sensors (such as inertial measurement units and surface electromyography sensors). After wavelet transform denoising, the atlas is constructed using a spatiotemporal feature map, where nodes represent joint / muscle groups and edge weights represent the strength of coordinated movement associations. The atlas is used to quantify the topological structure of a user's motor dysfunction, providing a dynamic spatiotemporal benchmark for multimodal feature fusion and resolving the disconnect between local features and global movement patterns in traditional assessment methods.

[0119] The motion pattern feature vector is a time-series feature vector extracted from the user's historical rehabilitation data. Specifically, a bidirectional long short-term memory (Bi-LSTM) network is used to encode parameters such as joint angular velocity, movement amplitude, and completion time of the user's motion trajectory, capturing the periodicity and abnormal patterns of the user's movement habits. The motion pattern feature vector represents the user's personalized motor ability baseline and serves as the query vector for the multi-head cross-attention mechanism, guiding the inheritance and optimization of the user's historical behavior patterns when generating virtual scene parameters.

[0120] The spatial layout feature vector is a 128-dimensional dense vector generated by topologically modeling geometric parameters in the virtual reality environment, such as obstacle positions, support surface inclination, and interactive object size, using a 3D point cloud encoder (PointNet++). The spatial layout feature vector encodes the physical constraints of the virtual scene and serves as a key-value pair for the attention mechanism to retrieve, ensuring that the generated rehabilitation plan is consistent with the dynamic feasibility of the environment.

[0121] The Dual-Delayed Deep Deterministic Policy Gradient (DDPG) algorithm is an improved reinforcement learning algorithm that consists of two independent critic networks and a delayed-update actor network. It optimizes the policy by minimizing the target Q-value difference (Bellman error) and policy smoothing regularization (adding truncated noise ε). It is designed to efficiently search for Pareto optimal solutions in the high-dimensional continuous space of virtual scene parameters. The dual critic networks suppress Q-value overestimation, addressing the policy oscillation and failure issues of traditional DDPG algorithms in complex rehabilitation tasks.

[0122] The action execution reward function is a dynamic reward function that combines the muscle activation coefficient a(t) and the mechanical feedback threshold τ. It is used to adjust the mechanical interaction intensity of the virtual scene through real-time feedback of muscle activation signals, maximizing the rehabilitation training effect while ensuring exercise safety, and breaking through the bottleneck of insufficient personalized adaptation caused by traditional static threshold settings.

[0123] The dynamic constraint on the mechanical feedback threshold dynamically adjusts the reaction force threshold applied by the virtual reality device through a proportional-integral controller based on the real-time changes in the muscle activation coefficient. It is used to automatically reduce the mechanical feedback intensity when the user's muscles fatigue or compensatory movements occur to prevent sports injuries. At the same time, the integral term eliminates steady-state errors and ensures a smooth transition of training intensity.

[0124] Specifically, the motor function assessment map is input into the encoder branch of the multimodal large model, where the encoder branch contains a multi-head cross-attention mechanism to generate a joint embedding representation through the following steps: the motion pattern feature vector in the user's historical rehabilitation data is used as the query vector Q, the spatial layout feature vector in the environmental parameters is used as the key vector K and the value vector V, and the scaled dot product is used to calculate the cross-modal attention weight. The specific formula is:

[0125]

[0126] Among them, d K is the dimension of the key vector, the motion pattern feature vector contains the temporal encoding of the user's historical motion trajectory, and the spatial layout feature vector extracts the geometric topological structure of the virtual scene through the 3D point cloud encoder. K and V are weightedly fused according to the attention weight to generate a dimensionally aligned joint embedding representation.

[0127] Based on the joint embedding representation, a double-delayed deep deterministic policy gradient algorithm is used to perform policy search in the virtual reality scene parameter space. Specifically, a dual critic network is constructed, where the first critic network and the second critic network receive the concatenation vector of the joint embedding representation and the virtual scene parameters respectively, and the network parameters are updated by minimizing the Bellman error. The target value y is calculated as follows:

[0128]

[0129] Where γ is the discount factor, ε is the Gaussian noise truncated to the interval [-c, c], r is the output value of the action execution reward function; θ' represents the parameters of the target Critic network; φ' represents the parameters of the target Actor network; s' represents the new state obtained by the environment transfer after the current state s performs action a; θ i ' represents the parameters of the i-th target critic network (i = 1 or 2), which is used to calculate the Q value of the next state s'; π φ' (s') represents the action generated by the target Actor network based on s'; the min operation means taking the smaller of the two target Critic network Q values ​​to suppress the overestimation of the Q value.

[0130] The action execution reward function dynamically constrains the mechanical feedback threshold τ based on the muscle activation coefficient a(t) in the physiological indicator data. The specific formula is:

[0131] τ=τ base ·σ(α·(1-|a(t)-a safe |));

[0132] Among them, τ base is the initial threshold; σ is the Sigmoid function; α is the trainable scaling factor; a safe Activates safe values ​​for preset muscles.

[0133] The policy network is optimized through a delayed update mechanism. Specifically, after every d Critic network updates, the policy gradient is calculated using the minimum Q value output by the dual Critic network. The update formula is:

[0134]

[0135] in, represents the gradient of the policy objective function; J(φ) represents the optimization objective function of the policy network, which represents the expected value of the long-term cumulative reward under the policy parameter φ. Its goal is to maximize the user's action execution reward (a dynamic function constrained by muscle activation and mechanical feedback) by adjusting the virtual scene parameters; Indicates the partial derivative of the parameter φ, which means updating the policy network parameters by the gradient ascent method to maximize J(φ); E s～D is the expected calculation of the experience replay pool, D is the experience replay pool, which stores historical state-action-reward samples; s contains the user's motion function evaluation map, environmental spatial layout characteristics, etc.; a is the action vector output by the policy network; s~D represents the state s randomly sampled from D, which is used to break the temporal correlation of data and improve training stability; represents the gradient of the policy network to the parameters; π φ (s) represents the policy network, which inputs state s, outputs action a, and has parameter φ.

[0136] In an optional manner, the interactive feedback module 230 is specifically configured to:

[0137] The eye tracking unit and force feedback gloves of the mixed reality device are used to collect the user's visual gaze hotspot data and joint torque response data in the virtual training scene to form biomechanical response data;

[0138] Using the dynamic time warping algorithm, the joint torque response data is aligned frame by frame with the expected mechanical trajectory in the personalized rehabilitation plan. The cumulative integral value of the joint torque deviation is calculated and used as the initial action execution deviation coefficient.

[0139] According to the heat weight of the visual gaze hotspot data, the initial action execution deviation coefficient is spatially weighted to generate the action execution deviation coefficient.

[0140] In this embodiment, the dynamic time warping algorithm is an algorithm used to align time series data of different lengths or time axis offsets. It calculates the minimum cumulative distance path by constructing a cost matrix to achieve nonlinear alignment, and is used to align the user's actual joint torque response with the expected trajectory frame by frame, eliminating errors caused by differences in movement speed and ensuring the timing consistency of deviation calculations.

[0141] Spatial attention weighting is a calculation method based on the deep learning attention mechanism. It assigns weights to the deviations of different areas through the spatial distribution of visual gaze hotspots (such as users pay more attention to the upper limb movement area). It is used to enhance the deviation coefficient weight of the area where the user's attention is focused (such as hand movement deviation), reduce interference from non-attention areas, and make the evaluation more in line with the user's actual perception of the rehabilitation focus.

[0142] Specifically, the eye tracking unit of the mixed reality device is used to capture the spatial coordinate sequence of the user's gaze point in the virtual scene at a preset sampling frequency, and the three-dimensional torque vector data of the specified joint is synchronously collected through the inertial measurement unit of the force feedback glove. The area in the spatial coordinate sequence where the dwell time exceeds the threshold is defined as the visual gaze hotspot data, and the three-dimensional torque vector data includes the torque amplitude and direction cosines of the selected joint, which together constitute the multimodal biomechanical response data.

[0143] Based on the three-dimensional torque vector data and the preset expected mechanical trajectory, the dynamic time warping algorithm is used to construct the deviation matrix between the observed torque vector and the expected torque vector. The optimal alignment path is searched through dynamic programming and the cumulative integral value along the path is calculated to generate the initial action execution degree deviation coefficient, where the cumulative integral value represents the timing alignment deviation between the actual action and the expected action.

[0144] A heat weight matrix is ​​generated according to the spatial distribution characteristics of visual gaze hotspot data. The initial action execution deviation coefficient is spatially weighted based on this matrix. The heat weight matrix uses the center of the gaze hotspot as the center of the Gaussian kernel and performs weighted mapping on the spatial projection area of ​​the interactive object in the virtual scene. Finally, the action execution deviation coefficient that integrates the visual attention characteristics is output.

[0145] Furthermore, the dynamic time warping algorithm specifically includes calculating the Euclidean distance between the observed torque vector and the expected torque vector of each frame to construct a deviation matrix, obtaining the alignment path with the minimum cumulative deviation through recursive path search, and taking the sum of the deviation value of each node in the path and the corresponding time difference as the cumulative integral value, where the time difference is dynamically adjusted according to the time warping ratio of the actual action and the expected action.

[0146] The generation of the heat weight matrix specifically includes Gaussian kernel density estimation of visual gaze hotspot data, calculating the weight value of each spatial coordinate point in the virtual scene, and point-by-point multiplication of it with the three-dimensional spatial projection function of the interactive object to generate a spatial weighting coefficient consistent with the user's visual attention distribution.

[0147] The inertial measurement unit of the force feedback glove uses a nine-degree-of-freedom sensor to collect three-dimensional torque data of the thumb, index finger, and middle finger joints. The expected mechanical trajectory is dynamically generated based on the physical properties and motion constraints of the interactive objects in the virtual scene.

[0148] The spatial projection function obtains the three-dimensional spatial coordinates of the interactive object in the user's field of view in real time through the depth sensor of the mixed reality device, and maps it into a two-dimensional plane projection area to calculate the spatial coverage of visual attention weighting.

[0149] In an optional manner, the adaptive adjustment module 240 is specifically configured to:

[0150] The action execution degree deviation coefficient is compared with the preset convergence threshold. If the convergence threshold is not reached, a fast adaptation algorithm based on model-agnostic meta-learning in the meta-learning framework is used to update the sub-network mask of the encoder weight matrix of the multimodal large model to obtain the target sub-network mask.

[0151] Through the parameter server in the distributed computing nodes, the target sub-network mask and the decoder weights of the multimodal large model are asynchronously gradient aggregated to generate the incrementally optimized target model parameters;

[0152] Update the multimodal large model based on the target model parameters to generate an incrementally optimized rehabilitation plan;

[0153] According to the decreasing rate of the action execution deviation coefficient in the historical iteration rounds, the value of the convergence threshold is dynamically adjusted, and the target model parameters are synchronized to the decoder weight storage unit of the multimodal large model until the target rehabilitation plan is output.

[0154] In this embodiment, the action execution deviation coefficient is the dynamic difference between the user's physiological indicators (such as the muscle activation coefficient) and the environmental interaction feedback. It is used to quantify the deviation between the user's action execution and the expected goal during rehabilitation training and serves as the core feedback signal for model optimization. By dynamically adjusting the model parameters, the mechanical feedback of the virtual scene is ensured to be adapted to the user's physiological state, avoiding sports injuries.

[0155] The fast adaptation algorithm of Model-Agnostic Meta-Learning (MAML) is a meta-learning framework that uses a two-layer optimization (the inner layer optimizes task-specific parameters, and the outer layer optimizes global initialization parameters) to enable the model to quickly adapt to new tasks with a small number of samples. In the scenario, the inner layer optimizes the sub-network mask update for the current user's rehabilitation task, and the outer layer optimizes the global gradient aggregation through a distributed parameter server. It is used to address the problem of insufficient generalization ability of traditional rehabilitation models due to individual differences in users. By dynamically adjusting the sub-network mask of the encoder weight matrix, the general feature extraction ability of the multimodal large model is retained, while quickly adapting to the user's personalized needs (such as adjusting the virtual scene rendering accuracy ρ).

[0156] Sub-network mask updates introduce a binary mask matrix into the neural network, dynamically freezing or activating specific neurons and updating only the sub-network parameters marked as "trainable" by the mask. Mask updates are based on the action execution deviation coefficient and generate a mask distribution through a meta-learning strategy. This is used to prevent "catastrophic forgetting" (i.e., performance degradation on old tasks) caused by global parameter updates during incremental optimization. For example, the ability to encode the spatial layout features of historical users is retained, while only fine-tuning the encoder weights related to the motion patterns of new users.

[0157] Parameter servers and asynchronous gradient aggregation are core components of the distributed computing architecture, responsible for storing model parameters and coordinating gradient synchronization across multiple nodes. The parameter server receives subnet mask gradients from different compute nodes and generates global incremental optimization parameters through weighted averaging (e.g., the AllReduce algorithm). This is used to address communication bottlenecks during the training of large-scale multimodal models, such as the CLIP ViT encoder. Asynchronous aggregation reduces inter-node gradient transmission latency while ensuring the stability of decoder weight updates.

[0158] Incremental optimization and dynamic convergence threshold adjustment adaptively adjust the convergence threshold based on the rate of decrease of the action execution deviation coefficient in historical iterations. This is used to balance model optimization speed and stability. Initially, a higher threshold is set to accelerate convergence, and later, the threshold is gradually reduced to improve parameter fine-tuning accuracy and avoid overfitting (for example, optimization oscillation of the curvature radius of the motion trajectory).

[0159] Specifically, the adaptive adjustment module 240 performs the following steps:

[0160] Based on the numerical relationship between the preset convergence threshold and the action execution degree deviation coefficient of the current iteration round, it is determined whether the model update condition is triggered. When the action execution degree deviation coefficient is greater than or equal to the convergence threshold, the encoder weight matrix of the multimodal large model is input into the fast adaptation algorithm unit of the meta-learning framework.

[0161] The fast adaptation algorithm unit is based on a model-agnostic meta-learning architecture. It generates the sub-network sensitivity distribution by calculating the Fisher information of each channel in the encoder weight matrix, and calculates the mask vector based on the sub-network sensitivity distribution and the loss function gradient of the encoder weight matrix to obtain the target sub-network mask.

[0162] The target sub-network mask and the decoder weight matrix of the multimodal large model are input into the parameter server of the distributed computing nodes. The decoder weight matrix is ​​incrementally updated by aggregating the gradient data of each computing node and combining the L2 regularization constraint to generate the target model parameters.

[0163] The decoder component of the multimodal large model is updated based on the target model parameters to generate an incrementally optimized rehabilitation plan that includes the rendering accuracy parameters of the virtual training scene, the curvature radius of the motion trajectory, and the feedback intensity parameters of the interactive device.

[0164] According to the exponentially weighted average decrease rate of the action execution degree deviation coefficient in historical iteration rounds, the value of the convergence threshold is dynamically adjusted. When the action execution degree deviation coefficient is less than the adjusted convergence threshold, the target model parameters are synchronized to the decoder weight storage unit, and the target rehabilitation plan is output.

[0165] In an optional manner, the system further includes a constraint optimization module; the constraint optimization module is used to:

[0166] The personalized rehabilitation plan is input into the pre-trained generative adversarial network discriminator to determine the feasibility score;

[0167] If the feasibility score is lower than the preset safety threshold, joint range of motion inequality constraints are imposed on the parameters of motion trajectory planning to regenerate a personalized rehabilitation plan that meets biomechanical safety conditions.

[0168] In this embodiment, the generative adversarial network (GAN) consists of a generator and a discriminator. The discriminator is a pre-trained neural network module that is used to distinguish whether the input data is a real sample (such as a rehabilitation plan designed by an expert) or a generated sample (such as the currently generated personalized rehabilitation plan). The discriminator is used as a "feasibility evaluator". By analyzing the motion trajectory parameters of the rehabilitation plan (such as joint torque and range of motion), the discriminator outputs a feasibility score to quantify whether the plan meets biomechanical safety and clinical effectiveness. For example, if the generated plan may cause joint overload or abnormal movement pattern, the discriminator will give a low score.

[0169] Joint range of motion inequality constraints are mathematical restrictions based on the range of motion of human joints. They are typically expressed as inequalities, limiting parameters such as joint angles and torques to ensure that the motion trajectory remains within the anatomically safe range. The constraint optimization module applies these constraints when the discriminator detects that a solution is insufficiently feasible. For example, an upper limit on knee flexion angle can be set to prevent cartilage wear caused by excessive flexion during rehabilitation training. These constraints can be implemented using dynamic programming or optimization algorithms to replan trajectory parameters.

[0170] Biomechanical safety conditions refer to the mechanical criteria that must be met during rehabilitation training, including joint force thresholds (such as avoiding exceeding the cartilage tolerance pressure), muscle activation balance (such as the ratio of antagonistic muscle co-contraction), and kinematic stability (such as the center of gravity offset limit during the gait cycle). Through the discriminator's scoring feedback, the system dynamically adjusts the program parameters (such as reducing training intensity or adjusting the movement trajectory) to ensure that rehabilitation movements do not cause secondary injuries (such as ligament strain or joint overload). For example, shear force limits are imposed on lumbar spine rehabilitation programs to prevent intervertebral disc damage.

[0171] The feasibility score is a quantitative indicator output by the discriminator, which comprehensively evaluates the compliance of the rehabilitation program in terms of biomechanics, clinical indications, and patient tolerance. The score range is usually 0-1, with 1 indicating complete feasibility. By presetting a safety threshold (such as 0.7), the system automatically triggers the optimization process. For example, if the score is lower than the threshold, the adductor muscles may be overstretched due to the excessive hip abduction angle in the program, and the trajectory needs to be replanned.

[0172] Preset safety thresholds are critical values ​​set based on clinical experience or big data statistics and are used to determine the safety of rehabilitation programs. The thresholds may be adjusted dynamically (e.g., based on the patient's stage of rehabilitation or real-time physiological data). Preset safety thresholds serve as triggers for constrained optimization. For example, in early rehabilitation after knee surgery, the threshold may be set to a low value (e.g., 0.6) to restrict the range of motion; as recovery progresses, the threshold is gradually increased.

[0173] Specifically, the feasibility of motion trajectories in personalized rehabilitation plans is evaluated through a pre-trained generative adversarial network discriminator. The discriminator is trained based on a three-dimensional joint torque trajectory dataset containing normal motion patterns and abnormal motion patterns. The normal motion pattern is constructed by collecting the three-dimensional torque vectors of multiple joints under standard movements, and the abnormal motion pattern is generated by applying random perturbations to the torque vectors.

[0174] A feasibility score of the personalized rehabilitation plan is calculated, and constrained optimization is triggered when the score is lower than a preset threshold, where the score indicates how well the movement trajectory matches the normal movement pattern.

[0175] An optimization model was constructed based on joint range of motion constraints and dynamic mechanical feedback thresholds. The joint kinematic parameters that met biomechanical safety conditions were solved using a quadratic programming algorithm. The joint range of motion constraints included limiting the angle range of specified joints, and the dynamic mechanical feedback threshold was dynamically adjusted based on the integral result of the real-time muscle activation degree deviation.

[0176] The optimized joint kinematic parameters are re-input into the generative adversarial network discriminator for iterative evaluation until the feasibility score reaches the preset qualification threshold and the final solution is output.

[0177] Furthermore, the training of the generative adversarial network discriminator specifically includes the use of a Wasserstein distance optimization framework with gradient penalty, which improves the recognition accuracy of abnormal motion patterns by constraining the Lipschitz continuity of the discriminator. The gradient penalty term is implemented by calculating the square difference between the gradient norm of randomly sampled data points and the target value.

[0178] The construction of joint motion constraint conditions specifically includes setting the maximum flexion angle of the shoulder joint to no more than 120 degrees and the minimum pronation angle of the elbow joint to no less than 15 degrees, and combining the physical properties of the interactive objects in the virtual training scene to generate a dynamically changing mechanical feedback threshold, where the mechanical feedback threshold is calculated based on the linear combination of the real-time deviation value of the muscle activation degree and its time integration result.

[0179] The objective function of the quadratic programming algorithm is set to minimize the square of the Euclidean distance between the optimized motion trajectory and the original trajectory. The constraints include the joint angle inequality constraint matrix and the physical safety limit of the maximum mechanical feedback value, where the physical safety limit is set to 0.8 times the preset optimal mechanical value.

[0180] The qualified threshold of the feasibility score is set to 0.75, and the optimization process is terminated early when the standard deviation of the scores of three consecutive evaluations during the iterative optimization process is less than 0.02.

[0181] Figure 2 FIG1 shows a flow chart of an embodiment of a method for personalized rehabilitation of peripheral nerve injury based on a multimodal large model provided by the present invention, as shown in FIG1 . Figure 2 As shown, the following steps are included:

[0182] The acquired multimodal time series data is dynamically weighted and fused through a spatiotemporal attention mechanism to extract cross-modal correlation feature vectors, and a motor function evaluation map is constructed based on an improved graph convolutional network.

[0183] The motor function assessment map is input into a pre-trained multimodal large model. Combined with the user's historical rehabilitation data and environmental parameters, a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning, and mechanical feedback thresholds is generated through a reinforcement learning strategy.

[0184] The system uses mixed reality devices to render virtual training scenes of personalized rehabilitation programs in real time, simultaneously collects the user's biomechanical response data in the virtual training scenes, and calculates the user's action execution deviation coefficient based on the dynamic time warping algorithm;

[0185] Based on the action execution deviation coefficient, a meta-learning framework is used to incrementally optimize the weight parameters of the multimodal large model to generate an incrementally optimized rehabilitation plan. The model parameters of the multimodal large model are then iterated online through distributed computing nodes until the target rehabilitation plan is output.

[0186] The technical solution of this embodiment improves the cross-modal correlation accuracy between motion characteristics and mechanical parameters, realizes dynamic matching of training scenarios and user motion capabilities, optimizes the virtual environment and mechanical feedback thresholds by dynamically adjusting rehabilitation program parameters, and shortens the program optimization cycle based on the online iterative optimization mechanism, solving the core defects of personalized adaptation lag and low efficiency of multimodal data utilization.

[0187] For the above-mentioned parameters and steps in the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model of this embodiment to realize the corresponding functions, please refer to the parameters and modules in the embodiment of the personalized rehabilitation system 200 for peripheral nerve injury based on a multimodal large model above, and will not be repeated here.

[0188] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330. The at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the aforementioned personalized rehabilitation systems for peripheral nerve injuries based on a multimodal large model. Specifically:

[0189] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320 to enable the electronic device 300 to implement any of the personalized rehabilitation methods for peripheral nerve injury based on a multimodal large model provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0190] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, which is loaded and executed by a processor so that the computer implements any of the above-mentioned personalized rehabilitation methods for peripheral nerve injury based on a multimodal large model.

[0191] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0192] In an exemplary embodiment, a computer program product or computer program is also provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the aforementioned methods for personalized rehabilitation of peripheral nerve injuries based on a multimodal large model.

[0193] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and to define a specific order or precedence. Where appropriate, the order used for similar objects may be interchanged, such that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0194] Those skilled in the art will appreciate that the present invention may be implemented as a system, method, or computer program product. Therefore, the present disclosure may be implemented in the following forms: entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or in a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention may be implemented in the form of a computer program product embodied in one or more computer-readable media containing computer-readable program code.

[0195] Any combination of one or more computer-readable media can be used. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0196] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A personalized rehabilitation system for peripheral nerve injury based on a multimodal large model, characterized by: include: Feature fusion module, personalized generation module, interactive feedback module and adaptive adjustment module; The feature fusion module is used to: dynamically weighted fuse the acquired multimodal time series data through a spatiotemporal attention mechanism, extract cross-modal correlation feature vectors, and construct a motor function evaluation map based on an improved graph convolutional network; The personalized generation module is used to: input the motor function assessment map into a pre-trained multimodal large model, combine the user's historical rehabilitation data and environmental parameters, and generate a personalized rehabilitation plan including virtual reality scene parameters, motion trajectory planning, and mechanical feedback thresholds through a reinforcement learning strategy; The interactive feedback module is used to: render a virtual training scene of the personalized rehabilitation program in real time through a mixed reality device, synchronously collect biomechanical response data of the user in the virtual training scene, and calculate the user action execution deviation coefficient based on a dynamic time warping algorithm; The adaptive adjustment module is used to: according to the action execution degree deviation coefficient, use the meta-learning framework to incrementally optimize the weight parameters of the multimodal large model, generate an incrementally optimized rehabilitation plan, and iterate the model parameters of the multimodal large model online through distributed computing nodes until the target rehabilitation plan is output.

2. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 1 is characterized in that: Also includes: Data acquisition module; the data acquisition module is used to: Acquire motion posture data through a multi-axis inertial sensor array; acquire plantar pressure data through a three-dimensional pressure sensing device; and acquire physiological index data through a bioelectric signal acquisition unit. The motion posture data, the plantar pressure data and the physiological index data are time synchronized and modally aligned to generate the multimodal time series data.

3. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 2, characterized in that: The feature fusion module is specifically used for: Inputting the motion posture data and the plantar pressure data into the temporal convolution branch and the spatial graph attention branch of the spatiotemporal attention mechanism respectively to generate a temporal dimension feature map and a spatial correlation weight matrix; Performing dynamic channel weighted fusion on the time dimension feature map according to the spatial correlation weight matrix to generate a cross-modal correlation feature vector; The cross-modal correlation feature vector is input into the graph convolutional network, and the motor function assessment map containing quantitative indicators of the degree of motor function impairment is output; wherein the adjacency matrix of the graph convolutional network is dynamically updated according to the physiological constraint relationship of the user's joint motion chain.

4. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 3 is characterized in that: The personalized generation module is specifically used for: Inputting the motor function assessment atlas into the encoder branch of the multimodal large model, fusing the motor pattern feature vector in the user's historical rehabilitation data and the spatial layout feature vector in the environmental parameters through a multi-head cross-attention mechanism to generate a joint embedding representation; Based on the joint embedding representation, a double-delayed deep deterministic policy gradient algorithm is used to search for a strategy that maximizes the action execution reward function in the virtual reality scene parameter space to generate the personalized rehabilitation plan; wherein, the action execution reward function dynamically constrains the mechanical feedback threshold according to the muscle activation degree coefficient in the physiological indicator data.

5. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 4 is characterized in that: The interactive feedback module is specifically used for: The eye tracking unit and force feedback gloves of the mixed reality device are used to collect the user's visual gaze hotspot data and joint torque response data in the virtual training scene to form the biomechanical response data; Using a dynamic time warping algorithm, the joint torque response data is aligned frame by frame with the expected mechanical trajectory in the personalized rehabilitation program, the cumulative integral value of the joint torque deviation is calculated, and the cumulative integral value is used as the initial action execution degree deviation coefficient; According to the heat weight of the visual gaze hotspot data, the initial action execution degree deviation coefficient is spatially weighted to generate the action execution degree deviation coefficient.

6. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 5, characterized in that: The adaptive adjustment module is specifically used for: Comparing the action execution degree deviation coefficient with a preset convergence threshold, and if the convergence threshold is not reached, using a fast adaptation algorithm based on model-agnostic meta-learning in the meta-learning framework to perform a sub-network mask update on the encoder weight matrix of the multimodal large model to obtain a target sub-network mask; Performing asynchronous gradient aggregation on the target subnetwork mask and the decoder weights of the multimodal large model through a parameter server in the distributed computing node to generate incrementally optimized target model parameters; Updating the multimodal large model based on the target model parameters to generate an incrementally optimized rehabilitation plan; According to the decreasing rate of the action execution degree deviation coefficient in the historical iteration rounds, the value of the convergence threshold is dynamically adjusted, and the target model parameters are synchronized to the decoder weight storage unit of the multimodal large model until the target rehabilitation plan is output.

7. The personalized rehabilitation system for peripheral nerve injury based on a multimodal large model according to claim 6, characterized in that: Also includes: Constrained optimization module; the constrained optimization module is used to: Inputting the personalized rehabilitation plan into a pre-trained generative adversarial network discriminator to determine a feasibility score; If the feasibility score is lower than a preset safety threshold, joint range of motion inequality constraints are applied to the parameters of the motion trajectory planning to regenerate the personalized rehabilitation plan that meets biomechanical safety conditions.

8. A personalized rehabilitation method for peripheral nerve injury based on a multimodal large model, characterized by: A personalized rehabilitation system for peripheral nerve injury based on a multimodal large model as described in any one of claims 1 to 7 is used.

9. An electronic device, characterized in that: The electronic device includes a processor, which is coupled to a memory. The memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model as described in claim 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by the processor so that the computer-readable storage medium implements the personalized rehabilitation method for peripheral nerve injury based on a multimodal large model as described in claim 8.

Images