Orthopedic rehabilitation-oriented multi-modal data fusion and real-time monitoring method and system

Abstract

The invention relates to the technical field of medical rehabilitation, and discloses a multi-modal data fusion and real-time monitoring method and system for orthopedic rehabilitation, and the system comprises a multi-modal data collection module, a data fusion processing module and a real-time monitoring execution module. The system is provided with a multi-modal feature calibration module, when the rehabilitation state of an orthopedic patient is dynamically evaluated, the system aligns the time sequence difference between a motion track and muscle fatigue features in real time by deploying a feature fusion model based on an attention mechanism, and joint load analysis distortion caused by multi-modal data delay can be corrected in a millisecond level; the accuracy of rehabilitation risk assessment is ensured, the joint secondary injury risk is reduced, an environment coupling risk identification unit is arranged, when abnormal action active early warning is carried out, the joint motion range and environment factors are dynamically associated through a reinforcement learning algorithm, the gait safety threshold value under the ground wet and slippery condition is corrected in real time, and the joint secondary injury risk is reduced. And the safety and the response real-time performance of the rehabilitation process are ensured.

Description

Technical Field

[0001] This invention relates to the field of medical rehabilitation technology, specifically to a multimodal data fusion and real-time monitoring method and system for orthopedic rehabilitation. Background Technology

[0002] With the increasing global aging trend, the incidence of orthopedic diseases continues to rise. Orthopedic treatments such as fractures and joint replacement surgery are gradually becoming an important part of patient rehabilitation. Orthopedic rehabilitation is a systematic recovery process for patients with injuries to or after surgery of the musculoskeletal system, including bones, muscles, joints, and ligaments. The core goal of orthopedic rehabilitation is to promote the functional recovery of bones and joints, reduce the occurrence of complications, and improve the quality of life of patients.

[0003] Currently, because the orthopedic rehabilitation monitoring process requires the integration of multi-source heterogeneous data, the traditional analysis system deployed can only independently process motion trajectory and physiological parameters when dynamically assessing the patient's rehabilitation status. It cannot correct the temporal misalignment between multimodal features in real time. When there is a millisecond-level delay between motion capture data and muscle fatigue features, it will cause distortion in joint load analysis and cannot guarantee the accuracy of rehabilitation risk assessment.

[0004] Therefore, a multimodal data fusion and real-time monitoring method and system for orthopedic rehabilitation is proposed to solve the above problems. Summary of the Invention

[0005] (a) Technical problems to be solved

[0006] To address the shortcomings of existing technologies, this invention provides a multimodal data fusion and real-time monitoring method and system for orthopedic rehabilitation, solving the problems mentioned in the background section.

[0007] (II) Technical Solution

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method and system for multimodal data fusion and real-time monitoring in orthopedic rehabilitation, wherein the method includes the following steps:

[0009] S1. Collect multimodal data on orthopedic rehabilitation, including motion capture data, physiological parameter data, and environmental sensor data;

[0010] S2. Based on the motion capture data, the physiological parameter data, and the environmental sensing data, perform data preprocessing to generate standardized motion data, standardized physiological data, and standardized environmental data;

[0011] S3. Perform multimodal data fusion processing based on the standardized motion data, the standardized physiological data, and the standardized environmental data, and generate fused rehabilitation feature data using a multimodal fusion model based on an attention mechanism;

[0012] S4. Perform real-time risk assessment processing based on the fused rehabilitation feature data, and generate rehabilitation risk assessment data using a reinforcement learning-driven risk prediction algorithm. When the risk is below the threshold, directly execute step S6.

[0013] S5. When the risk exceeds the threshold, based on the fused rehabilitation feature data and the rehabilitation risk assessment data, abnormal intervention strategy generation processing is performed to generate personalized rehabilitation intervention strategy data.

[0014] S6. Based on the fused rehabilitation feature data and the personalized rehabilitation intervention strategy data, perform rehabilitation progress visualization processing to construct a real-time rehabilitation monitoring report;

[0015] S7. Based on the real-time rehabilitation monitoring report, perform feedback output processing to generate rehabilitation optimization suggestion data.

[0016] Preferably, step S1 includes the following steps:

[0017] S11. Collect patient joint movement trajectory data through wearable motion sensors, including angle, velocity and acceleration parameters, and generate motion capture data;

[0018] S12. Collect patient physiological parameter data, including electromyography signals, heart rate and blood oxygen saturation parameters, through bioelectric sensors, and generate physiological parameter data;

[0019] S13. Collect rehabilitation environment data through environmental sensors, including temperature, humidity, light and ground stability parameters, and generate environmental sensing data.

[0020] Preferably, step S2 includes the following steps:

[0021] S21. Import the motion capture data, the physiological parameter data, and the environmental sensing data into the rehabilitation monitoring platform, and use wavelet transform algorithm for noise filtering and feature normalization to generate standardized motion data, standardized physiological data, and standardized environmental data.

[0022] Preferably, step S3 includes the following steps:

[0023] S31. Construct a multimodal fusion model based on the attention mechanism. The model inputs are the standardized motion data, the standardized physiological data, and the standardized environmental data, and the output is fused rehabilitation feature data.

[0024] S32. The attention-based multimodal fusion model adopts the Transformer architecture and calculates the weights of each modality using the attention weight formula:

[0025]

[0026] Where α m h represents the attention weights for the m-th modality. ref The feature vector of the standardized motion data, dimension 128, h m Let be the feature vector of the m-th mode, with a dimension of 128, where m = 1, 2, 3 correspond to motion, physiology, and environment, and M is the total number of modes, with a value of 3.

[0027] When α m If the value is less than 0.2, the modality data is discarded and a re-acquisition is triggered;

[0028] S33. When the feature vector dimensions are inconsistent, a fully connected layer is used for dimension alignment to ensure that the fused rehabilitation feature data is consistent.

[0029] Preferably, step S4 includes the following steps:

[0030] S41. Obtain the fusion rehabilitation feature data;

[0031] S42. Using a reinforcement learning risk prediction algorithm, the Q-value update formula is:

[0032]

[0033] Where s t Let a be the fused feature vector at time t, with dimension 128. t Let a be the risk level of the action at time t. t ∈{1,2,3,4,5}, where 5 represents the highest risk, r t r represents the negative value of the recovery progress deviation. t ∈[-1,1], η is the learning rate of 0.01, γ is the discount factor of 0.9, and Q(s,a) is the state-action value function;

[0034] S43. When the predicted risk value is lower than the preset threshold, output the rehabilitation risk assessment data as low risk and directly execute step S6.

[0035] S44. When the predicted risk value is higher than the preset threshold, the output rehabilitation risk assessment data is high risk.

[0036] Preferably, step S5 includes the following steps:

[0037] S51. When the rehabilitation risk assessment data is high-risk, a decision tree algorithm is used to generate a personalized intervention strategy, wherein the decision tree splitting criterion uses the information gain formula:

[0038]

[0039] Where D is the current node dataset, A is the splitting feature, V is the number of splitting values ​​of feature A (less than 5), Entropy(D) is the information entropy of dataset D (threshold [0,1]), and Gain is the information gain of feature A (greater than 0.05). v To represent a subset of data that takes the value v on attribute A;

[0040] S52. Generate personalized rehabilitation intervention strategy data and update the rehabilitation monitoring platform database.

[0041] Preferably, step S6 includes the following steps:

[0042] S61. Input the fused rehabilitation feature data and the personalized rehabilitation intervention strategy data into the visualization engine to generate a real-time rehabilitation monitoring report;

[0043] S62. The real-time rehabilitation monitoring report includes a 3D motion trajectory map, a physiological trend curve, and a risk heat map.

[0044] Preferably, step S7 includes the following steps:

[0045] S71. The real-time rehabilitation monitoring report is sent to the terminal device via a wireless transmission module;

[0046] S72. Trigger alarms, update rehabilitation plans, and provide remote medical consultations based on the rehabilitation optimization suggestion data.

[0047] Preferably, the training method for the attention-based multimodal fusion model includes:

[0048] S91. Collect historical rehabilitation datasets, including samples of exercise, physiological, and environmental data;

[0049] S92. Use the cross-entropy loss function and Adam optimizer to train the model, with the number of iterations set to 1000 rounds;

[0050] S93. To prevent overfitting, an early stopping mechanism is used, and training is stopped when the accuracy on the validation set reaches 95% or higher.

[0051] Preferably, the system includes a multimodal data acquisition module, a data fusion processing module, and a real-time monitoring execution module;

[0052] The multimodal data acquisition module includes a motion data acquisition unit, a physiological data acquisition unit, and an environmental data acquisition unit;

[0053] The motion data acquisition unit collects joint motion data through wearable sensors; the physiological data acquisition unit collects physiological parameter data through bioelectric sensors; and the environmental data acquisition unit collects rehabilitation environment data through environmental sensors.

[0054] The data fusion processing module includes a data preprocessing unit, a multimodal fusion unit, and a risk assessment unit;

[0055] The data preprocessing unit performs filtering and standardization processing based on motion, physiological and environmental data; the multimodal fusion unit generates fusion feature data using an attention-based fusion model; and the risk assessment unit uses a reinforcement learning algorithm for real-time risk prediction.

[0056] The real-time monitoring execution module includes a visualization report generation unit and a feedback output unit;

[0057] The visualization report generation unit generates a monitoring report based on fusion features and intervention strategies, and the feedback output unit triggers alarms and plan updates based on optimization suggestion data.

[0058] (III) Beneficial Effects

[0059] Compared with existing technologies, this invention provides a multimodal data fusion and real-time monitoring method and system for orthopedic rehabilitation, which has the following beneficial effects:

[0060] 1. In this invention, the system sets up a multimodal feature calibration module. When performing dynamic assessment of the rehabilitation status of orthopedic patients, the system deploys a feature fusion model based on attention mechanism to real-time align the temporal differences between motion trajectory and muscle fatigue features. This enables millisecond-level correction of joint load analysis distortion caused by multimodal data delay, ensuring the accuracy of rehabilitation risk assessment and reducing the risk of secondary joint injury.

[0061] 2. In this invention, by setting up an environmental coupling risk identification unit, when actively warning of abnormal movements, the system dynamically associates joint mobility with environmental factors through reinforcement learning algorithms, and corrects the gait safety threshold under slippery ground conditions in real time. This enables the system to actively identify joint-environment coupling risks, reduce the misjudgment rate of traditional single threshold judgment, and trigger intervention strategies in real time when the environment changes abruptly, ensuring the safety and real-time response of the rehabilitation process.

[0062] 3. In this invention, by setting up an individualized decision update module, when generating rehabilitation intervention strategies, the system synchronously integrates the patient's age and bone density characteristics through dynamic information entropy calculation, and corrects the key feature splitting criteria in real time. This enables the system to adaptively match the rehabilitation needs of different patients, avoids strategy deviations caused by static rules, and further improves the feasibility of intervention plans and the accurate control of rehabilitation processes. Attached Figure Description

[0063] Figure 1 This is a flowchart of the multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to the present invention;

[0064] Figure 2 This is a diagram illustrating the architecture of the multimodal data fusion and real-time monitoring system for orthopedic rehabilitation, as described in this invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] For specific implementation examples, please refer to: Figure 1-2 A method and system for multimodal data fusion and real-time monitoring in orthopedic rehabilitation, comprising the following steps:

[0067] S1. Collect multimodal data on orthopedic rehabilitation, including motion capture data, physiological parameter data, and environmental sensor data;

[0068] Principle: The system achieves heterogeneous data acquisition through a multimodal data acquisition module, including a motion data acquisition unit, a physiological data acquisition unit, and an environmental data acquisition unit. The motion data acquisition unit uses wearable motion sensors to capture joint motion trajectory data in real time, with a sampling frequency of up to 100Hz, ensuring the updating of angle, velocity, and acceleration parameters. The physiological data acquisition unit uses bioelectric sensors, including surface electromyography (sEMG) and photoplethysmography (PPG), to acquire electromyographic signals, heart rate, and blood oxygen saturation, and transmits them via Bluetooth 5.0. The environmental data acquisition unit deploys temperature and humidity sensors and ground stability sensors to monitor environmental factors. All units start synchronously, and data timing consistency is ensured through timestamp alignment.

[0069] Technical benefits: This solution enables real-time parallel acquisition of multi-source data, reduces data latency, and improves the accuracy of subsequent fusion.

[0070] S2. Based on motion capture data, physiological parameter data, and environmental sensor data, perform data preprocessing to generate standardized motion data, standardized physiological data, and standardized environmental data;

[0071] Principle: The data preprocessing unit of the data fusion processing module uses wavelet transform algorithm for noise filtering: Formula In this process, scale parameter 'a' controls the frequency resolution, and translation parameter 'b' determines the time window, separating high-frequency noise from low-frequency signals. Subsequently, the Z-score normalization formula is used. The feature values ​​are mapped to the interval [-1,1], where μ is the mean of the dataset and σ is the standard deviation. This unit achieves fully automated processing on the rehabilitation monitoring platform.

[0072] Technical effects: Wavelet transform filters out environmental interference noise, normalization eliminates the differences in dimensions between different sensors, and ensures the comparability of multimodal data.

[0073] S3. Perform multimodal data fusion processing based on standardized exercise data, standardized physiological data, and standardized environmental data, and use an attention-based multimodal fusion model to generate fused rehabilitation feature data;

[0074] Principle: A multimodal fusion unit constructs a Transformer model based on an attention mechanism. The input is standardized data with 128 dimensions. Attention weight formula. In the middle, h ref h is the feature vector of motion data. m For each modal feature, m = 1, 2, 3 corresponds to motion, physiology, and environment, respectively. When the weight α m A re-acquisition mechanism is triggered when the value is less than 0.2. When feature dimensions are inconsistent, alignment is achieved through a fully connected layer y = Wx + b, and the weight matrix W is optimized to 128×128. The model is trained using historical datasets, with the cross-entropy loss function and Adam optimizer iterated for 1000 rounds.

[0075] Technical benefits: This solution aligns the temporal differences between motion trajectories and muscle fatigue characteristics in real time, corrects data latency, and reduces the distortion rate of joint load analysis. An attention mechanism dynamically allocates weights, improving the reliability of fused features.

[0076] S4. Real-time risk assessment processing is performed based on fused rehabilitation feature data. A reinforcement learning-driven risk prediction algorithm is used to generate rehabilitation risk assessment data. When the risk is below the threshold, step S6 is executed directly.

[0077] Principle: The risk assessment unit employs a reinforcement learning algorithm, with the Q-value update formula... In the middle, state s t To fuse feature vectors, dimension 128, action a t ∈{1,2,3,4,5} represents the risk level, with 5 being the highest risk and the reward being r. t = -|Actual Progress - Expected Progress| Quantifies rehabilitation deviation, range [-1,1], state transition probability P(s) t+1 |s t ,a t The model environment is dynamic. The learning rate η = 0.01, and the discount factor γ = 0.9. When the predicted risk value is lower than the preset threshold of 2, a low-risk output is given; otherwise, a high-risk alarm is triggered.

[0078] Technical effects: Reinforcement learning dynamically correlates joint range of motion with environmental factors, and corrects risk predictions in real time. System response time is shortened, misjudgment rate is reduced, and rehabilitation safety is ensured under sudden environmental changes.

[0079] S5. When the risk exceeds the threshold, abnormal intervention strategy generation processing is performed based on the fusion of rehabilitation feature data and rehabilitation risk assessment data to generate personalized rehabilitation intervention strategy data.

[0080] Principle: When the risk assessment is high, the system uses a decision tree algorithm to generate a strategy. Information gain formula. In the dataset D, fused features and individual patient data are included, such as age and bone mineral density. Splitting features A select key factors, including joint range of motion or ground stability. The number of split values ​​V is less than 5, and the information gain threshold is greater than 0.05. The decision tree dynamic splitting criterion integrates patient features and adjusts the intervention strategy in real time.

[0081] Technical Effects: This solution overcomes the limitations of static rules by adaptively matching patient needs through an individualized decision-making update module. It shortens strategy generation time, improves the feasibility of intervention plans, enhances the accuracy of rehabilitation process control, and avoids secondary injuries caused by strategy deviations.

[0082] S6. Based on the integrated rehabilitation feature data and personalized rehabilitation intervention strategy data, perform visualization processing of rehabilitation progress and construct a real-time rehabilitation monitoring report;

[0083] Principle: The visualization report generation unit is based on fusion features and intervention strategies, using a 3D visualization engine to dynamically map data: joint motion trajectories are converted into 3D skeletal models, physiological trend curves are drawn in real time, and risk heatmaps are correlated with environmental factors. The report supports multi-view collaborative display and achieves accurate updates.

[0084] Technical benefits: This solution provides interactive real-time monitoring, allowing medical staff to simultaneously observe abnormalities, including gait instability areas and environmental coupling risks, reducing visualization delays, improving the speed of rehabilitation process regulation and response, and enhancing clinical decision-making efficiency.

[0085] S7. Process feedback output based on real-time rehabilitation monitoring reports to generate rehabilitation optimization suggestion data;

[0086] Principle: The feedback output unit sends reports to the terminal device via a wireless transmission module, and triggers an optimization suggestion data mechanism: alarms alert the patient via buzzer and LED; rehabilitation plan updates automatically adjust device parameters; remote medical consultation initiates video consultation, forming a closed-loop feedback system.

[0087] Technical effects: This solution ensures real-time implementation of intervention strategies and minimizes the risk of secondary injury. Multi-terminal linkage improves real-time response, enhances overall system security, and increases rehabilitation optimization rate.

[0088] S1 includes the following steps:

[0089] S11. Collect patient joint movement trajectory data through wearable motion sensors, including angle, velocity and acceleration parameters, and generate motion capture data;

[0090] S12. Collect patient physiological parameter data, including electromyography signals, heart rate and blood oxygen saturation parameters, through bioelectric sensors, and generate physiological parameter data;

[0091] S13. Collect rehabilitation environment data through environmental sensors, including temperature, humidity, light and ground stability parameters, and generate environmental sensing data.

[0092] S2 includes the following steps:

[0093] S21. Import motion capture data, physiological parameter data and environmental sensor data into the rehabilitation monitoring platform, and use wavelet transform algorithm to perform noise filtering and feature normalization to generate standardized motion data, standardized physiological data and standardized environmental data.

[0094] S22. Feature normalization processing uses the Z-score normalization formula:

[0095]

[0096] Where x is the original feature value, μ is the mean of the dataset, and σ is the standard deviation. norm These are the normalized eigenvalues, with the range normalized to [-1, 1], to ensure the comparability of data at different scales;

[0097] S23. The noise filtering formula for the wavelet transform algorithm is:

[0098]

[0099] Where C(a,b) are wavelet coefficients, a is the scaling parameter controlling the frequency resolution, b is the translation parameter controlling the time position, f(t) is the original signal, and ψ * Let dt be the complex conjugate of the mother wavelet function, and let dt be the time derivative. For time scaling and panning operations, This is the normalization factor.

[0100] S3 includes the following steps:

[0101] S31. Construct a multimodal fusion model based on the attention mechanism. The model input consists of standardized motion data, standardized physiological data, and standardized environmental data, and the output consists of fused rehabilitation feature data.

[0102] S32. The attention-based multimodal fusion model adopts the Transformer architecture and calculates the weights of each modality using the attention weight formula:

[0103]

[0104] Where α m h represents the attention weights for the m-th modality. ref The feature vector of the standardized motion data, dimension 128, h m Let be the feature vector of the m-th mode, with a dimension of 128, where m = 1, 2, 3 correspond to motion, physiology, and environment, and M is the total number of modes, with a value of 3.

[0105] When α m If the value is less than 0.2, the modality data is discarded and a re-acquisition is triggered;

[0106] S33. When the feature vector dimensions are inconsistent, a fully connected layer is used for dimension alignment to ensure that the fused rehabilitation feature data is consistent.

[0107] S34. The formula for dimensional alignment of fully connected layers is:

[0108] y = Wx + b;

[0109] Where x is the input feature vector, W is the weight matrix, b is the bias vector, and y is the output feature vector, with a fixed dimension of 128.

[0110] S4 includes the following steps:

[0111] S41. Obtain fusion rehabilitation feature data;

[0112] S42. Using a reinforcement learning risk prediction algorithm, the Q-value update formula is:

[0113]

[0114] Where s t Let a be the fused feature vector at time t, with dimension 128. t Let a be the risk level of the action at time t. t ∈{1,2,3,4,5}, where 5 represents the highest risk, r t r represents the negative value of the recovery progress deviation. t ∈[-1,1], η is the learning rate of 0.01, γ is the discount factor of 0.9, and Q(s,a) is the state-action value function;

[0115] S43. When the predicted risk value is lower than the preset threshold, output the rehabilitation risk assessment data as low risk and directly execute step S6.

[0116] S44. When the predicted risk value is higher than the preset threshold, the output rehabilitation risk assessment data is high risk.

[0117] S45. The formula for the state transition probability of the reinforcement learning algorithm is:

[0118] P(s t+1 |s t ,a t );

[0119] Where P is the probability distribution, s t Let a be the current state vector, with dimension 128. t For the current action, s t+1 This is the next state vector;

[0120] S46. The reward function formula is:

[0121] r t = -|Actual Progress - Expected Progress|;

[0122] The actual progress is calculated in real time from sensor data, while the expected progress is set based on the rehabilitation plan. t The range is [-1, 1], and the negative sign indicates a deviation penalty.

[0123] S5 includes the following steps:

[0124] S51. When the rehabilitation risk assessment data indicates a high risk, a decision tree algorithm is used to generate a personalized intervention strategy, where the decision tree splitting criterion uses the information gain formula:

[0125]

[0126] Where D is the current node dataset, A is the splitting feature, V is the number of splitting values ​​of feature A (less than 5), Entropy(D) is the information entropy of dataset D (threshold [0,1]), and Gain is the information gain of feature A (greater than 0.05). v To represent a subset of data that takes the value v on attribute A;

[0127] S52. Generate personalized rehabilitation intervention strategy data and update the rehabilitation monitoring platform database.

[0128] S6 includes the following steps:

[0129] S61. Input the integrated rehabilitation feature data and personalized rehabilitation intervention strategy data into the visualization engine to generate a real-time rehabilitation monitoring report;

[0130] S62. Real-time rehabilitation monitoring reports include 3D motion trajectory maps, physiological trend curves, and risk heat maps.

[0131] S7 includes the following steps:

[0132] S71. Send the real-time rehabilitation monitoring report to the terminal device via the wireless transmission module;

[0133] S72. Trigger alarms, rehabilitation plan updates, and remote medical consultations based on rehabilitation optimization suggestion data.

[0134] Training methods for attention-based multimodal fusion models include:

[0135] S91. Collect historical rehabilitation datasets, including samples of exercise, physiological, and environmental data;

[0136] S92. Use the cross-entropy loss function and Adam optimizer to train the model, with the number of iterations set to 1000 rounds;

[0137] S93. To prevent overfitting, an early stopping mechanism is used, and training is stopped when the accuracy on the validation set reaches 95% or higher.

[0138] The system includes a multimodal data acquisition module, a data fusion and processing module, and a real-time monitoring and execution module;

[0139] The multimodal data acquisition module includes a motion data acquisition unit, a physiological data acquisition unit, and an environmental data acquisition unit;

[0140] The motion data acquisition unit collects joint motion data through wearable sensors; the physiological data acquisition unit collects physiological parameter data through bioelectric sensors; and the environmental data acquisition unit collects rehabilitation environment data through environmental sensors.

[0141] The data fusion processing module includes a data preprocessing unit, a multimodal fusion unit, and a risk assessment unit;

[0142] The data preprocessing unit performs filtering and standardization on motion, physiological and environmental data; the multimodal fusion unit generates fused feature data using an attention-based fusion model; and the risk assessment unit uses reinforcement learning algorithms for real-time risk prediction.

[0143] The real-time monitoring execution module includes a visualization report generation unit and a feedback output unit;

[0144] The visualization report generation unit generates monitoring reports based on fusion features and intervention strategies, while the feedback output unit triggers alarms and plan updates based on optimization suggestion data.

[0145] The system operates as follows:

[0146] During the multimodal data acquisition phase, the system collects three key types of data in real time through a heterogeneous sensor network:

[0147] Motion capture data: Wearable sensors capture joint angle, velocity, and acceleration parameters to quantify joint range of motion and movement patterns;

[0148] Principle: A wearable inertial measurement unit (IMU) is used to capture joint three-dimensional motion parameters, including angle, velocity, and acceleration, in real time at a sampling frequency of 100Hz. A timestamp synchronization protocol ensures reduced alignment errors of multi-sensor data.

[0149] Technical benefits: Improves the accuracy of knee joint motion trajectory capture, providing a high-fidelity data foundation for dynamic evaluation.

[0150] Physiological parameter data: Electromyographic signals, heart rate, and blood oxygen saturation are acquired through bioelectric sensors to reflect the degree of muscle fatigue and cardiovascular load;

[0151] Principle: The physiological data acquisition unit uses an sEMG sensor to capture electromyographic signals at a sampling rate of 200Hz. The muscle fatigue spectrum is analyzed by fast Fourier transform. The PPG sensor monitors heart rate and blood oxygen saturation, and the time synchronization accuracy is shortened.

[0152] Technical benefits: Enables real-time correlation between muscle fatigue and exercise status, improves the accuracy of cardiovascular load monitoring, and avoids the risk of overtraining.

[0153] Environmental sensing data: Deploy temperature, humidity, light, and ground stability sensors to monitor physical risk factors in the rehabilitation environment;

[0154] Principle: The environmental data acquisition unit uses a piezoelectric sensor to detect ground stability with a sensitivity of 0.1g, and a temperature and humidity sensor with an accuracy of 0.5℃ to dynamically sample environmental factors.

[0155] Technical benefits: Real-time identification of environmental changes, shortening of risk warnings and improving the safety of the rehabilitation environment.

[0156] This stage ensures the comprehensiveness and real-time nature of the data source, providing raw input for subsequent analysis;

[0157] Principle: The multimodal data acquisition module starts all units in parallel and achieves global synchronization of timestamps through the NTP protocol, with a sampling interval of 10ms.

[0158] Technical effects: Reduced data acquisition latency, improved consistency of multi-source data, laying the foundation for dynamic evaluation, and reduced joint trajectory distortion rate.

[0159] During the data preprocessing and standardization stage, noise and dimensional differences in the raw data need to be eliminated.

[0160] Wavelet transform denoising: using the continuous wavelet transform formula High-frequency noise and low-frequency features are separated, with scale parameter a controlling the frequency resolution and translation parameter b determining the time window;

[0161] Principle: The data preprocessing unit uses Morlet wavelet basis functions to adaptively adjust a and b to match the signal characteristics, and retains the effective frequency band after filtering.

[0162] Technical benefits: Improves noise suppression rate and signal-to-noise ratio, ensures accuracy of joint load analysis, and reduces the risk of data distortion.

[0163] Z-score normalization: according to the formula Each modal data is mapped to the [-1,1] interval to eliminate sensor range differences and output standardized motion, physiological and environmental data, ensuring the comparability of multi-source data;

[0164] Principle: Normalization is based on real-time calculation of the mean μ and standard deviation σ, dynamically adapting to the data distribution.

[0165] Technical effects: Data from different sensors are scaled uniformly, the consistency of input to the fusion model is improved, and the false positive rate is reduced.

[0166] In the multimodal feature fusion stage, the Transformer model based on the attention mechanism achieves temporal alignment and weight allocation:

[0167] Attention weight calculation: using motion data as the reference feature vector, according to the formula... Dynamically allocate weights for motion, physiological, and environmental modalities, when α m If the value is greater than 0.2, retain it; otherwise, trigger a re-collection.

[0168] Principle: The multimodal fusion unit uses a Transformer encoder layer, the score function calculates cosine similarity, and the weights are dynamically adjusted.

[0169] Technical effects: Corrects timing misalignment, improves the accuracy of joint load analysis, and reduces the risk of secondary injury.

[0170] Dimension alignment: When the feature vector dimensions are inconsistent, the output dimension is unified to 128 through the fully connected layer formula y=Wx+b to avoid information loss;

[0171] Principle: The weights W of the fully connected layer are pre-trained and optimized, the input dimension is adaptively matched, and the output is fixed at 128 dimensions.

[0172] Technical effects: Improved feature information retention rate, enhanced fusion efficiency, and guaranteed reliability of real-time risk assessment.

[0173] This stage addresses the temporal misalignment issue caused by millisecond-level latency in multimodal data, generating a fused rehabilitation feature vector s. t .

[0174] In the real-time risk assessment and early warning phase, reinforcement learning models drive dynamic risk prediction:

[0175] Q-value iterative update: based on the state-action value function Where state s t To fuse feature vectors, action a t ∈{1,2,3,4,5}, where 5 represents the highest risk and the reward is r. t = -|Actual progress - Expected progress| Quantitative rehabilitation deviation;

[0176] Principle: The risk assessment unit adopts the Q-learning algorithm. The state space is based on fused features, the action space corresponds to the risk level, and the reward feedback is calculated in real time.

[0177] Technical benefits: Shortens prediction response time, reduces misjudgment rate under sudden environmental changes, and ensures the safety of the rehabilitation process.

[0178] Environmental coupling correction: Correlate joint mobility with environmental parameters, dynamically adjust gait safety thresholds, and trigger a high-risk alarm when the predicted risk value exceeds the preset threshold; otherwise, proceed to the visualization process.

[0179] Principle: Real-time cross-analysis of joint mobility and environmental parameters, including ground humidity and stability level, establishes a motion-environment risk matrix. When slippery ground is detected and the friction coefficient is less than 0.4, the gait safety threshold is automatically lowered by 30%. By integrating sudden changes in temperature and humidity with joint load data, a composite risk index is generated to achieve multi-factor collaborative early warning.

[0180] Technical effects: Reduced misjudgment rate in sudden environmental changes, shortened response time for risk warnings of falls on slippery surfaces, reduced incidence of joint-environment coupled injuries, and reduced frequency of rehabilitation interruptions due to environmental interference.

[0181] In the personalized intervention strategy generation phase, adaptive solutions are generated for high-risk scenarios:

[0182] Dynamic splitting of decision trees: based on information gain criterion Integrating individual patient characteristics, an intervention branch is generated when the splitting value V is less than 5 and the gain value is greater than 0.05;

[0183] Principle: Real-time calculation of information entropy value, dynamic selection of key splitting features, including joint mobility and ground stability, and generation of intervention branches based on information gain threshold.

[0184] Technical effects: Improved strategy matching, reduced secondary damage rate, and shortened decision response time.

[0185] The strategy is updated in real time: the output plan includes training intensity adjustment, anti-slip measures instructions and remote consultation suggestions, and updates the platform database;

[0186] Principle: A four-dimensional intervention matrix is ​​dynamically constructed, including training intensity, anti-slip measures, remote consultation, and equipment parameters. The parameters are adaptively adjusted based on the risk level and written to the database in real time via SQL to ensure that the strategy takes effect immediately.

[0187] Technical effects: Increased success rate of anti-slip measures activation in wet and slippery environments, improved accuracy of rehabilitation equipment parameter adjustment, shortened strategy execution delay, and reduced risk of secondary injury.

[0188] This stage breaks through the limitations of static rules, enabling accurate intervention that is adaptive to the patient;

[0189] Principle: Breaking through the limitations of static rules, the system automatically adjusts safety thresholds and intervention strategies based on changes in the patient's joint mobility, bone density, and environmental factors through a dynamic decision tree and a real-time weight correction engine, thereby achieving adaptive and precise control of the rehabilitation process.

[0190] Technical effects: It achieves dynamic matching of individualized patient needs, reduces the deviation rate of rehabilitation strategies, reduces the incidence of secondary injuries in elderly patients, improves the precision of intervention, and meets the real-time response needs of orthopedic rehabilitation.

[0191] In the rehabilitation progress visualization phase, a 3D visualization engine builds interactive reports:

[0192] Dynamic data mapping: transforming fused features and intervention strategies into 3D joint motion trajectories, physiological indicator trend curves, and environmental risk heatmaps;

[0193] Principle: The system uses inverse kinematics algorithm to convert joint motion trajectories into 3D skeletal models in real time, uses dynamic Bézier curves to fit the trend of physiological parameter changes, maps environmental parameters to RGBA color space to generate risk heat maps, and uses a multi-view collaborative engine to achieve 50Hz data refresh.

[0194] Technical effects: Reduced visualization latency, improved recognition rate of abnormal joint movements, shortened early warning response time of environmental risk heatmaps, and improved efficiency of medical and nursing decision-making.

[0195] Multi-view collaborative display: Medical staff can simultaneously observe gait abnormalities and related environmental factors, and the report provides visual feedback with millisecond latency, supporting real-time adjustment of the rehabilitation process;

[0196] Principle: Based on the priority of rehabilitation stage, the view space is dynamically allocated to realize the automatic layout engine. The joint movement trajectory and the risk heat map coordinates are synchronized in real time to establish a cross-view association mechanism. Abnormal physiological parameters automatically trigger the dynamic migration of the highlight focus of the corresponding view.

[0197] Technical effects: Improved efficiency in multi-dimensional data cognition by medical staff and increased risk identification rate of joint-environment coupling, resulting in shorter clinical decision-making time.

[0198] In the closed-loop feedback and execution phase, the system forms a "monitoring-decision-execution" closed loop:

[0199] Multi-terminal linkage: Send alarms to patient terminals via wireless transmission, and push risk reports and remote consultation requests to physician terminals;

[0200] Principle: A three-level response protocol is established: high-risk alarm → patient terminal: vibration + LED red light, rehabilitation update → medical staff terminal: pop-up window, remote consultation → cloud video channel, using the MQTT lightweight transmission protocol to achieve millisecond-level command broadcasting, and the device status is fed back to the central control console in real time to form a closed-loop verification.

[0201] Technical effects: Alarm transmission delay is reduced, cross-terminal command synchronization success rate is improved, remote consultation connection response time is compressed, and system fault tolerance is improved.

[0202] Active execution mechanism: Automatically adjusts rehabilitation equipment parameters, updates training plans, and activates safety devices based on optimization suggestions;

[0203] Principle: The device directly controls the parameters of rehabilitation equipment, including treadmill speed and safety cable tension, through the device API interface. It dynamically analyzes rehabilitation risk assessment data to trigger four-dimensional control: intensity adjustment, safety device activation, and environmental intervention, including automatic dehumidification, anti-slip, and emergency braking. The execution effect is pre-simulated based on a digital twin model.

[0204] Technical effects: Improved accuracy of equipment parameter adjustment, shortened activation time of safety devices, reduced incidence of secondary damage, and increased success rate of automatic environmental control.

[0205] This stage ensures the timely implementation of intervention strategies and minimizes the risk of secondary damage.

[0206] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0207] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal data fusion and real-time monitoring method for orthopedic rehabilitation, characterized by: The method includes the following steps: S1. Collect multimodal data on orthopedic rehabilitation, including motion capture data, physiological parameter data, and environmental sensor data; S2. Based on the motion capture data, the physiological parameter data, and the environmental sensing data, perform data preprocessing to generate standardized motion data, standardized physiological data, and standardized environmental data; S3. Perform multimodal data fusion processing based on the standardized motion data, the standardized physiological data, and the standardized environmental data, and generate fused rehabilitation feature data using a multimodal fusion model based on an attention mechanism; S4. Perform real-time risk assessment processing based on the fused rehabilitation feature data, and generate rehabilitation risk assessment data using a reinforcement learning-driven risk prediction algorithm. When the risk is below the threshold, directly execute step S6. S5. When the risk exceeds the threshold, based on the fused rehabilitation feature data and the rehabilitation risk assessment data, abnormal intervention strategy generation processing is performed to generate personalized rehabilitation intervention strategy data. S6. Based on the fused rehabilitation feature data and the personalized rehabilitation intervention strategy data, perform rehabilitation progress visualization processing to construct a real-time rehabilitation monitoring report; S7. Based on the real-time rehabilitation monitoring report, perform feedback output processing to generate rehabilitation optimization suggestion data.

2. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S1 includes the following steps: S11. Collect patient joint movement trajectory data through wearable motion sensors, including angle, velocity and acceleration parameters, and generate motion capture data; S12. Collect patient physiological parameter data, including electromyography signals, heart rate and blood oxygen saturation parameters, through bioelectric sensors, and generate physiological parameter data; S13. Collect rehabilitation environment data through environmental sensors, including temperature, humidity, light and ground stability parameters, and generate environmental sensing data.

3. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S2 includes the following steps: S21. Import the motion capture data, the physiological parameter data, and the environmental sensing data into the rehabilitation monitoring platform, and use wavelet transform algorithm for noise filtering and feature normalization to generate standardized motion data, standardized physiological data, and standardized environmental data.

4. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S3 includes the following steps: S31. Construct a multimodal fusion model based on the attention mechanism. The model inputs are the standardized motion data, the standardized physiological data, and the standardized environmental data, and the output is fused rehabilitation feature data. S32. The attention-based multimodal fusion model adopts the Transformer architecture and calculates the weights of each modality using the attention weight formula: Where α m h represents the attention weights for the m-th modality. ref The feature vector of the standardized motion data, dimension 128, h m Let be the feature vector of the m-th mode, with a dimension of 128, where m = 1, 2, 3 correspond to motion, physiology, and environment, and M is the total number of modes, with a value of 3. When α m If the value is less than 0.2, the modality data is discarded and a re-acquisition is triggered; S33. When the feature vector dimensions are inconsistent, a fully connected layer is used for dimension alignment to ensure that the fused rehabilitation feature data is consistent.

5. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S4 includes the following steps: S41. Obtain the fusion rehabilitation feature data; S42. Using a reinforcement learning risk prediction algorithm, the Q-value update formula is: Where s t Let a be the fused feature vector at time t, with dimension 128. t Let a be the risk level of the action at time t. t ∈{1,2,3,4,5}, where 5 represents the highest risk, r t r represents the negative value of the recovery progress deviation. t ∈[-1,1], η is the learning rate of 0.01, γ is the discount factor of 0.9, and Q(s,a) is the state-action value function; S43. When the predicted risk value is lower than the preset threshold, output the rehabilitation risk assessment data as low risk and directly execute step S6. S44. When the predicted risk value is higher than the preset threshold, the output rehabilitation risk assessment data is high risk.

6. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S5 includes the following steps: S51. When the rehabilitation risk assessment data is high-risk, a decision tree algorithm is used to generate a personalized intervention strategy, wherein the decision tree splitting criterion uses the information gain formula: Where D is the current node dataset, A is the splitting feature, V is the number of splitting values ​​of feature A (less than 5), Entropy(D) is the information entropy of dataset D (threshold [0,1]), and Gain is the information gain of feature A (greater than 0.05). v To represent a subset of data that takes the value v on attribute A; S52. Generate personalized rehabilitation intervention strategy data and update the rehabilitation monitoring platform database.

7. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S6 includes the following steps: S61. Input the fused rehabilitation feature data and the personalized rehabilitation intervention strategy data into the visualization engine to generate a real-time rehabilitation monitoring report; S62. The real-time rehabilitation monitoring report includes a 3D motion trajectory map, a physiological trend curve, and a risk heat map.

8. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: S7 includes the following steps: S71. The real-time rehabilitation monitoring report is sent to the terminal device via a wireless transmission module; S72. Trigger alarms, update rehabilitation plans, and provide remote medical consultations based on the rehabilitation optimization suggestion data.

9. The multimodal data fusion and real-time monitoring method for orthopedic rehabilitation according to claim 1, characterized in that: The training method for the attention-based multimodal fusion model includes: S91. Collect historical rehabilitation datasets, including samples of exercise, physiological, and environmental data; S92. Use the cross-entropy loss function and Adam optimizer to train the model, with the number of iterations set to 1000 rounds; S93. To prevent overfitting, an early stopping mechanism is used, and training is stopped when the accuracy on the validation set reaches 95% or higher.

10. A multimodal data fusion and real-time monitoring system for orthopedic rehabilitation, used to implement the multimodal data fusion and real-time monitoring method for orthopedic rehabilitation as described in any one of claims 1-9, characterized in that: The system includes a multimodal data acquisition module, a data fusion processing module, and a real-time monitoring and execution module; The multimodal data acquisition module includes a motion data acquisition unit, a physiological data acquisition unit, and an environmental data acquisition unit; The motion data acquisition unit collects joint motion data through wearable sensors; the physiological data acquisition unit collects physiological parameter data through bioelectric sensors; and the environmental data acquisition unit collects rehabilitation environment data through environmental sensors. The data fusion processing module includes a data preprocessing unit, a multimodal fusion unit, and a risk assessment unit; The data preprocessing unit performs filtering and standardization processing based on motion, physiological and environmental data; the multimodal fusion unit generates fusion feature data using an attention-based fusion model; and the risk assessment unit uses a reinforcement learning algorithm for real-time risk prediction. The real-time monitoring execution module includes a visualization report generation unit and a feedback output unit; The visualization report generation unit generates a monitoring report based on fusion features and intervention strategies, and the feedback output unit triggers alarms and plan updates based on optimization suggestion data.

Images