Multimodal vision neural network guided pulmonary rehabilitation motion correction system

Abstract

The present application relates to the technical field of lung rehabilitation, and discloses a multi-modal visual neural network guided lung rehabilitation action correction system. The system comprises nine functional modules. A visual data acquisition module first acquires a multi-modal visual data sequence of a patient's rehabilitation action, and a feature sequence extraction module extracts an action feature sequence therefrom. After a change point in the feature sequence is located by a change point identification module, a change intensity index calculation module generates a change intensity index in combination with information such as a feature change amplitude, and then obtains an average change intensity index. An action error index calculation module calculates an action error index by fusing a feature sequence distance and the average change intensity index, and then obtains a visual dynamic coefficient and a skeletal posture coefficient, respectively. Finally, a neural network parameter adjustment module optimizes network parameters accordingly, realizes precise correction of the rehabilitation action, and provides scientific and efficient rehabilitation guidance for the patient.

Description

Technical Field

[0001] This invention relates to the field of pulmonary rehabilitation technology, specifically to a pulmonary rehabilitation movement correction system guided by a multimodal visual neural network. Background Technology

[0002] Lung diseases, including chronic obstructive pulmonary disease (COPD), pulmonary fibrosis, and sequelae of pneumonia, are prevalent and their incidence is increasing globally, severely impacting patients' respiratory function and quality of life. Pulmonary rehabilitation, a crucial means of improving the prognosis of lung disease patients, effectively strengthens respiratory muscles, improves lung ventilation, and enhances exercise tolerance through scientific breathing exercises, postural drainage, and limb function training, helping patients gradually regain normal living abilities. However, the standardization of pulmonary rehabilitation movements directly determines the rehabilitation effect. Incorrect movements not only fail to achieve the expected rehabilitation goals but may even increase the patient's respiratory burden, causing discomfort such as chest tightness and shortness of breath, thus delaying the rehabilitation process.

[0003] Traditional pulmonary rehabilitation guidance primarily relies on face-to-face instruction and manual correction by professional medical staff. Medical staff demonstrate standard movements for patients to imitate and learn, observing their performance during training, pointing out errors and making adjustments as needed. However, this model has several limitations. First, high-quality medical resources are unevenly distributed, and the number of professional pulmonary rehabilitation physicians is relatively scarce, making it difficult to meet the personalized guidance needs of a large number of patients. This is especially true in remote areas, where patients often face the dilemma of "difficulty in accessing medical care, and even greater difficulty in obtaining rehabilitation guidance." Second, manual guidance is costly. Patients need to regularly travel to the hospital for training and guidance, increasing their time and travel costs, as well as their financial burden, making it difficult for some patients to persist with long-term rehabilitation training.

[0004] With the development of computer vision technology, some studies have attempted to apply it to the field of pulmonary rehabilitation movement monitoring, using cameras to capture images of patients' movements to achieve preliminary recognition of rehabilitation actions. However, most existing technologies use single-modal visual data for analysis, such as relying solely on two-dimensional image data. This approach is easily affected by factors such as shooting angle, lighting conditions, and background interference, leading to incomplete and inaccurate extraction of movement features. For example, when capturing patients' chest expansion movements, two-dimensional images cannot accurately capture changes in the anteroposterior diameter of the thoracic cavity and the three-dimensional movement trajectory of bones and joints, thus failing to fully reflect the standardization of the movement.

[0005] Existing systems lack scientific quantitative indicators and precise judgment logic for identifying movement errors. Most systems simply compare patient movements with standard movements in images to determine consistency, failing to analyze the specific type and severity of the error. Furthermore, the core algorithm model parameters of these systems are usually fixed, making it difficult to adaptively adjust to different patients' physical conditions and movement characteristics, resulting in poor versatility and adaptability. For example, elderly patients, due to weaker flexibility and muscle strength, exhibit significantly different amplitude and speed of rehabilitation movements compared to younger patients. Models with fixed parameters struggle to accurately identify movement errors in both groups, thus failing to provide effective correction guidance. Therefore, developing a system capable of accurately, comprehensively, and adaptively correcting pulmonary rehabilitation movements has become a pressing issue in the field of pulmonary rehabilitation. Summary of the Invention

[0006] The purpose of this invention is to provide a multimodal visual neural network-guided lung rehabilitation movement correction system to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides a multimodal visual neural network-guided lung rehabilitation movement correction system, the system comprising: The visual data acquisition module is configured to acquire multimodal visual data sequences of patients' pulmonary rehabilitation movements; The feature sequence extraction module is configured to extract action feature sequences based on multimodal visual data sequences. The change point recognition module is configured to identify change points from action feature sequences; The change intensity index calculation module is configured to obtain the change intensity index based on the feature change magnitude at the change point, the variation of the feature sequence, and the mean change of the local feature sequence. The module for obtaining the average change intensity index is configured to obtain the average change intensity index based on the change intensity index. The motion error index calculation module is configured to obtain the motion error index based on the distance between motion feature sequences and the average change intensity index. The visual dynamic coefficient calculation module is configured to obtain visual dynamic coefficients based on the action error index. The skeletal pose coefficient calculation module is configured to obtain skeletal pose coefficients based on pose angle information in action feature sequences. The neural network parameter adjustment module configures the parameters of the neural network based on visual dynamic coefficients and skeletal pose coefficients.

[0008] Preferably, the visual data acquisition module includes a multimodal visual sensor configured to simultaneously acquire RGB image sequences, depth image sequences, and infrared image sequences of the patient, wherein the multimodal visual sensor is arranged around the patient to cover the full range of rehabilitation movements.

[0009] Preferably, the feature sequence extraction module is configured to extract skeletal key point sequences as motion feature sequences from RGB image sequences, depth image sequences, and infrared image sequences, wherein the skeletal key point sequences include joint positions and motion trajectories.

[0010] Preferably, the change point identification module is configured to use a time series analysis algorithm to detect peaks and valleys in each action feature sequence as change points, wherein the time series analysis algorithm calculates the local extrema of feature values ​​based on a sliding window.

[0011] Preferably, the change intensity index calculation module is configured to, for each change point, calculate the absolute value of the difference between the feature value at the change point and the feature value at the previous moment as the forward change amount, obtain the change consistency amount based on the maximum value of the first-order difference of the action feature sequence, take a preset time window centered on the change point to calculate the absolute value of the difference between the mean of features before the change point and the mean of features after the change point within the window as the local change amount; and obtain the change intensity index based on the forward change amount, the change consistency amount, and the local change amount through a predefined weighting function.

[0012] Preferably, the average change intensity index acquisition module is configured to, for each time point, when the time point is identified as a change point in all action feature sequences, calculate the mean of the change intensity indices of all action feature sequences at that time point as the average change intensity index.

[0013] Preferably, the motion error index calculation module is configured to use a clustering algorithm to cluster the average change intensity index of all visual sensors to obtain clusters. For each visual sensor, the mean of the Euclidean distance between the local window sequence of its motion feature sequence and the local window sequence of the corresponding motion feature sequence of other visual sensors in the same cluster is calculated as the trend dissimilarity index. The motion error index is obtained by using a scaling function based on the average change intensity index and the trend dissimilarity index.

[0014] Preferably, the visual dynamic coefficient calculation module is configured to use the action error index of each visual sensor as input and a threshold segmentation algorithm to obtain the optimal segmentation threshold, mark time points where the action error index is greater than or equal to the optimal segmentation threshold as high error times, mark time points where the action error index is less than the optimal segmentation threshold as low error times, calculate the absolute value of the difference between the mean action error index at high error times and the mean action error index at low error times as the behavior difference index, calculate the ratio of the number of low error times to the total number of time points as the stationary duration ratio, and then obtain the visual dynamic coefficient by multiplying the mean action error index at high error times, the behavior difference index, and the stationary duration ratio.

[0015] Preferably, the skeletal attitude coefficient calculation module is configured to extract the pitch angle, roll angle and yaw angle at each time point from the action feature sequence for each visual sensor as attitude angle information, and calculate the average value of the attitude angle change rate between consecutive time points under high error time as the skeletal attitude coefficient.

[0016] Preferably, the neural network parameter adjustment module is configured to normalize the visual dynamic coefficients and skeletal pose coefficients, add the normalized visual dynamic coefficients and skeletal pose coefficients together and multiply them by a preset initial parameter value to obtain the adjustment value of the neural network parameters, which is used to update the weight matrix of the neural network.

[0017] Compared with the prior art, the beneficial effects of the present invention are: The visual data acquisition module employs a multimodal visual data acquisition approach, breaking through the limitations of traditional single-modal data. Multimodal visual data sequences can capture patients' rehabilitation movement information from different dimensions, including not only the appearance features of the movements but also the spatial trajectory and dynamic changes of skeletal motion, effectively avoiding information loss problems caused by angle deviations, lighting interference, and other factors inherent in single-data types. This comprehensive data acquisition mode provides rich raw material for subsequent feature extraction and movement analysis, making the description of movement features more complete and objective, laying the foundation for accurate identification of movement errors.

[0018] The feature sequence extraction module extracts motion feature sequences based on multimodal visual data sequences, realizing the transformation from raw data to effective information. This module can accurately capture key features in rehabilitation movements, including changes in the angle of limb joints, the speed and amplitude of movements, and the shift of the body's center of gravity. These features directly reflect the standardization and coordination of the movements. Compared with existing technologies that simply extract movement contour features, the feature sequences extracted by this system are more targeted and representative, and can deeply characterize the essential attributes of the movements, providing accurate analytical basis for subsequent identification of change points and error judgment.

[0019] The collaboration between the change point identification module and the change intensity index calculation module enables refined analysis of motion changes. Change point identification accurately locates key nodes where anomalies occur during the motion process, while the change intensity index calculation quantitatively assesses the degree of anomaly at the change point by comprehensively considering the amplitude of feature changes, the variability of feature sequences, and the mean change of local feature sequences. This quantitative analysis method changes the traditional system's approach of judging motion errors solely through subjective observation, making the judgment of motion anomalies more scientific and objective. It can effectively distinguish between minor motion deviations and serious motion errors, providing a clear direction for subsequent targeted corrections.

[0020] The integration of the average variation intensity index acquisition module and the movement error index calculation module constructs a scientific movement error assessment system. The movement error index comprehensively considers the distance between movement feature sequences and the average variation intensity index, reflecting both the overall difference between the patient's movement and the standard movement, as well as the degree of abnormality at key points during the movement. This multi-dimensional assessment method can comprehensively and accurately measure the overall situation of movement errors, avoiding the one-sidedness of a single assessment indicator. This allows medical staff and patients to clearly understand the specific circumstances of movement errors, providing a reliable basis for developing personalized rehabilitation training programs.

[0021] The calculation of visual dynamic coefficients and skeletal posture coefficients provides supplementary assessments of movement quality from different dimensions. Visual dynamic coefficients, generated based on a movement error index, reflect the dynamic standardization of movements; skeletal posture coefficients, obtained from postural angle information within the movement feature sequence, directly reflect the rationality of skeletal movement. These two factors complement each other, forming a comprehensive assessment of rehabilitation movements. This assessment not only identifies surface-level issues with movement standardization but also delves into potential errors at the skeletal posture level, providing more detailed reference information for movement correction.

[0022] The neural network parameter adjustment module adjusts the neural network parameters based on visual dynamic coefficients and skeletal posture coefficients, achieving adaptive optimization of the system. This module enables the neural network model to adjust its internal parameters in real time according to the different movement characteristics, physical conditions, and changes during the rehabilitation process of different patients, improving the model's adaptability to different individual movements. Compared with existing models with fixed parameters, the neural network model of this system has greater flexibility and versatility, and can provide personalized movement recognition and correction services for different patients' specific situations. It is especially suitable for special groups such as elderly patients and patients with weak physical functions, effectively expanding the system's applicability.

[0023] From a clinical application perspective, this system effectively reduces the workload of medical staff and addresses the shortage of professional rehabilitation guidance resources. Patients can use the system for independent rehabilitation training at home or in the community. The system can monitor movements in real time and provide precise corrective guidance, reducing patients' dependence on medical staff and improving the convenience and accessibility of rehabilitation training. Simultaneously, the system can record the entire rehabilitation training process, providing comprehensive data support for medical staff to assess rehabilitation effects and adjust training plans. This helps to achieve personalized and precise rehabilitation treatment and promotes the optimization and upgrading of pulmonary rehabilitation treatment models. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the working principle of the multimodal visual neural network-guided lung rehabilitation movement correction system described in this invention. Figure 2 A flowchart illustrating the operation of the change intensity index calculation module; Figure 3 A flowchart illustrating the operation of the visual dynamic coefficient calculation module; Figure 4 A diagram for monitoring and analyzing errors in pulmonary rehabilitation training movements; Figure 5 A graph showing adaptive adjustment of neural network parameters. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Please see Figure 1This invention provides a multimodal visual neural network-guided lung rehabilitation movement correction system. The system includes: a visual data acquisition module, a feature sequence extraction module, a change point identification module, a change intensity index calculation module, an average change intensity index acquisition module, a movement error index calculation module, a visual dynamic coefficient calculation module, a skeletal posture coefficient calculation module, and a neural network parameter adjustment module. The visual data acquisition module acquires multimodal visual data sequences of the patient's lung rehabilitation movements. The feature sequence extraction module processes the multimodal visual data sequences to generate movement feature sequences. The change point identification module analyzes the movement feature sequences to identify key change points. The change intensity index calculation module evaluates the feature change amplitude at the change points, the variation of the feature sequences, and the mean change of the local feature sequences to derive the change intensity index. The average change intensity index acquisition module integrates the change intensity indices to obtain the average change intensity index. The movement error index calculation module calculates the movement error index by combining the distance between movement feature sequences and the average change intensity index. The visual dynamic coefficient calculation module derives the visual dynamic coefficient based on the movement error index. The skeletal posture coefficient calculation module calculates the skeletal posture coefficient using the posture angle information in the movement feature sequences. The neural network parameter adjustment module adjusts the neural network parameters based on the visual dynamic coefficient and the skeletal posture coefficient. The system achieves real-time correction of pulmonary rehabilitation movements through the collaborative processing of multimodal visual data, thereby improving the accuracy and safety of rehabilitation training.

[0027] Example 1: The visual data acquisition module includes a multimodal visual sensor, which is positioned around the patient to cover the entire range of rehabilitation movements. The multimodal visual sensor is configured to simultaneously acquire RGB image sequences, depth image sequences, and infrared image sequences from the patient. The arrangement of the multimodal visual sensor follows spatial geometry optimization principles, with the arrangement scheme determined based on the spatial dimensions of the rehabilitation training area and the patient's range of motion. The multimodal visual sensor is mounted on a fixed bracket or adjustable track, and the overlapping viewpoints of the sensor array form a blind-spot-free monitoring network. RGB image sequence acquisition uses a high-resolution color camera with a frame rate adapted to the rapid movements of the human body. The RGB image sequence records the surface color information and texture changes of the patient during rehabilitation training. Depth image sequence acquisition is achieved through an active stereo vision camera or a time-of-flight camera. The depth image sequence contains distance data from each pixel in the scene to the camera, and the depth information is used to construct a three-dimensional spatial model of the movements. Infrared image sequence acquisition utilizes a thermal infrared sensor or near-infrared light source in conjunction with an infrared-sensitive camera. The infrared image sequence captures the distribution of human body thermal radiation or near-infrared reflection characteristics. Infrared data provides supplementary visual information when ambient light is insufficient or shadows are present. Synchronous acquisition by the multimodal vision sensor is achieved through a global shutter trigger signal or a precision hardware clock. Each frame of the RGB image sequence, depth image sequence, and infrared image sequence is accurately timestamped, with timestamp alignment accuracy down to the millisecond level, ensuring strict consistency of multimodal data in the temporal dimension. Data from the multimodal vision sensor is transmitted to the processing unit via a high-speed data interface using Gigabit Ethernet or USB 3.0 protocols, with transmission bandwidth sufficient to meet the real-time transmission requirements of multiple high-definition video streams.

[0028] The feature sequence extraction module is configured to extract skeletal keypoint sequences as motion feature sequences from RGB image sequences, depth image sequences, and infrared image sequences. These skeletal keypoint sequences include joint positions and motion trajectories. The processing flow of the feature sequence extraction module is based on a deep learning architecture, with a convolutional neural network (CNN) model serving as the backbone network for feature extraction. The CNN model accepts multimodal image data as input. RGB image sequences are input to the color channels of the CNN model, which extracts spatial feature maps from the RGB image sequences. These spatial feature maps are then used by a keypoint detection head to output a two-dimensional joint heatmap. Depth image sequences are preprocessed into standardized depth maps, which are then input to the depth branch of the CNN model. The depth branch outputs the three-dimensional coordinate offsets of the joints. Infrared image sequences undergo contrast enhancement processing, and the enhanced infrared images are input to the infrared branch of the CNN model. The infrared branch outputs auxiliary joint confidence scores based on thermal features. In the multimodal feature fusion stage of the CNN model, color features, depth features, and infrared features are concatenated into tensors. The fused feature tensors are then used to regress joint coordinates through fully connected layers. The joint position output is a set of 3D coordinates of the major human joints, including the shoulder, elbow, wrist, hip, knee, and ankle joints. The coordinates of each joint are defined relative to the world coordinate system or camera coordinate system. The motion trajectory generation module performs temporal smoothing on the joint position sequence, using Kalman or Bayesian filters to remove observation noise. The motion trajectory is represented as a 3D coordinate curve of each joint on the time axis. The skeletal keypoint sequence data structure includes timestamps, joint identifiers, and 3D coordinate values. The skeletal keypoint sequence is stored in a circular buffer or time-series database format, supporting real-time streaming processing. The computational optimization of the feature sequence extraction module is accelerated using a graphics processor, and the convolutional neural network model is deployed on a hardware platform with parallel computing capabilities, keeping inference latency within the video frame rate range.

[0029] The calibration process for the multimodal vision sensors establishes a spatial correspondence between the RGB camera, depth camera, and infrared camera. The calibration procedure uses a specially designed calibration board that appears simultaneously in the fields of view of all sensors. This calibration board has a pattern of marker points with known geometric dimensions. The internal parameters of the RGB camera, including focal length, principal point, and distortion coefficients, are calculated using the Zhang Zhengyou calibration method. The internal parameters of the depth camera, including depth scaling factor and projection matrix, are obtained using a plane fitting calibration method. The internal parameters of the infrared camera are calibrated based on a blackbody radiation reference source. The external parameters between the multimodal vision sensors are described by a rigid transformation matrix, which is obtained by solving a hand-eye calibration problem. The calibration accuracy is evaluated using reprojection error. The topology of the multimodal vision sensor arrangement considers the typical motion planes of rehabilitation movements. The sensor array covers three anatomical planes: sagittal, coronal, and transverse. The sensor viewing angle is adjusted to avoid limb occlusion. The color balance of the RGB image sequence is automatically adjusted based on a white balance algorithm, and the exposure time dynamically adapts to ambient light. The video encoding of the RGB image sequence uses H.264 compression to reduce storage space. Invalid value imputation of the depth image sequence is performed using a nearest neighbor interpolation algorithm. Depth skip edges are smoothed using a bilateral filter, and the measurement range of the depth image sequence matches the spatial scale of rehabilitation training. Non-uniformity correction of the infrared image sequence is based on a blackbody radiation calibration curve. The temperature sensitivity of the infrared image sequence meets the range of human surface temperature variations, and the frame integration time of the infrared image sequence is adaptively adjusted. The time synchronization mechanism of the multimodal visual data sequence adopts a precision time protocol. The main control unit sends periodic synchronization pulses to all sensors, and the sensor hardware trigger signal lines are directly connected to the clock output of the main control unit.

[0030] The convolutional neural network (CNN) model for feature sequence extraction is trained on a multimodal rehabilitation action dataset containing various rehabilitation action samples from healthy subjects and pulmonary rehabilitation patients. Data annotation was manually completed by rehabilitation medicine experts. Input data augmentation for the CNN model includes random rotation, scaling, and color jitter. The loss function for the multimodal input branch is designed as a weighted mean squared error loss, and the output dimension of the joint position regression head corresponds to the number of joints multiplied by three. The CNN model's network architecture utilizes either a residual network or a densely connected network structure. A feature pyramid network fuses multi-scale features to improve small joint detection accuracy, and an attention mechanism module enhances the feature response of key regions. Joint position post-processing includes bone length constraint verification, a physical constraint module to detect whether the inter-joint distance is within an anatomically reasonable range, and correction of abnormal joint positions using a kinematic model. The motion trajectory smoothing algorithm employs a physically constrained trajectory optimization, with the objective function minimizing joint acceleration and jerk. The temporal consistency of the motion trajectory is evaluated using a dynamic time warping algorithm. The real-time performance of the feature sequence extraction module is optimized through model quantization and pruning techniques. Floating-point operations in the convolutional neural network model are converted to fixed-point operations, reducing the number of model parameters while keeping accuracy loss within an acceptable range. The output interface for the skeletal keypoint sequence provides network stream or shared memory access methods. The consumer module for action feature sequences obtains data through a callback function mechanism, and data stream transmission latency is monitored using timestamp differential.

[0031] The fault tolerance mechanism of the multimodal vision sensor is designed with a redundant data acquisition mode. When a sensor fails, the system automatically switches to the backup sensor data stream, and the sensor health status is continuously monitored through a heartbeat mechanism. The degradation handling strategy of the feature sequence extraction module includes adjusting the fusion weights when the quality of a certain modality data deteriorates, reducing the weights of RGB image sequences under overexposure conditions, reducing the confidence level of depth image sequences in transparent object regions, and activating auxiliary judgment logic for infrared image sequences when the ambient temperature is close to body temperature. The adaptive adjustment of the multimodal vision sensor layout is based on automatic calculation of the patient's height and weight index. Sensor height and angle are automatically adjusted via a motorized pan-tilt head, and coverage testing uses standard action templates for verification. The cross-patient generalization ability of the feature sequence extraction module is improved through domain-adaptive training. The convolutional neural network model uses test-time adaptive technology during the inference phase, and batch normalized statistics are updated online to adapt to the new patient feature distribution. The coordinate system of the skeletal keypoint sequence is unified into the patient's body coordinate system, with the origin set at the pelvic center point, and the coordinate axis directions determined based on principal component analysis to determine the patient's main movement direction. The motion trajectory is represented by derived values ​​of joint angles, angular velocities, and angular accelerations. Dimensionality compression of the trajectory feature vector is achieved through principal component analysis, and the dimensionality reduction of the feature sequence preserves the discriminative information of the rehabilitation movements. The storage format of the multimodal visual data sequences adopts open standard protocols: RGB image sequences are stored in MPEG format, depth image sequences in point cloud data format, and infrared image sequences in radiation temperature matrix format. Metadata fields record acquisition parameters and patient information. The extended interface of the feature sequence extraction module supports the integration of third-party skeletal tracking algorithms. The algorithm plugin manager dynamically loads skeletal keypoint detection libraries from different vendors, and the abstract interface layer unifies data formats and calling specifications.

[0032] Example 2: See Figure 2The change point identification module is configured to use time series analysis algorithms to detect peaks and troughs in each action feature sequence as change points. The time series analysis algorithm calculates local extrema of feature values ​​based on a sliding window. Action feature sequences are input into the data buffer of the change point identification module. The data buffer uses a first-in, first-out queue to manage time-series data, and each action feature sequence is processed independently. The width of the sliding window is dynamically set according to the typical movement cycle of pulmonary rehabilitation movements. The movement cycle is determined by a preset movement frequency template by the rehabilitation physician, and the width of the sliding window is directly proportional to the movement cycle. The window sliding step size is set to one sampling time unit to ensure that each data point in the sequence has a chance to become the window center. The local extrema detection algorithm compares the feature value of the window center point with the feature values ​​of all adjacent points within the window. When the feature value of the window center point is greater than the feature values ​​of all adjacent points, the point is identified as a peak and marked as a change point; when the feature value of the window center point is less than the feature values ​​of all adjacent points, the point is identified as a trough and marked as a change point. The change point identification module outputs the timestamp index and type identifier of the change points. The list of change points is stored in a shared memory area for subsequent modules to read. The time series analysis algorithm includes a denoising preprocessing step. A moving average filter is applied to the original action feature sequence, and the filter window size and the extreme value detection window size are set independently. Normalization of the action feature sequence is completed before change point detection. Normalization maps feature values ​​of different dimensions to a range of zero to one, avoiding the influence of differences in feature value magnitudes on extreme value detection. The change point identification module operates at the same frequency as the sampling frequency of the action feature sequence; each new data point triggers a sliding window calculation, and real-time requirements are met through interrupt service routines.

[0033] The variation intensity index calculation module is configured to calculate the absolute value of the difference between the feature value at each change point and the feature value at the previous moment as the forward variation. It obtains the variation consistency value based on the maximum value of the first-order difference of the action feature sequence. Centered on the change point, it calculates the absolute value of the difference between the mean feature value before and after the change point within a preset time window as the local variation. The forward variation calculation uses the feature value corresponding to the change point's timestamp minus the feature value at the previous sampling moment; the absolute value of the difference is used to obtain the instantaneous variation amplitude. The variation consistency value calculation requires traversing the first-order difference values ​​of the entire action feature sequence. The first-order difference is obtained by subtracting the feature values ​​at adjacent moments, and the maximum value of all differences is used as a representation of the sequence variation degree. The width of the preset time window is independent of the sliding window width and is set based on the duration of the rehabilitation action phase. The window boundaries are truncated to prevent out-of-bounds errors. The mean value of features before the change point is calculated using the arithmetic mean of all feature values ​​before the change point within a preset time window. The mean value of features after the change point is calculated using the arithmetic mean of all feature values ​​after the change point within the preset time window. The absolute value of the difference between the two means reflects the local horizontal shift caused by the change point. The change intensity index calculation module integrates a predefined weighting function, which accepts three input parameters: forward change, change consistency, and local change. The weighting function adopts a linear weighted combination form, where each input parameter is multiplied by a weighting coefficient and then summed. The weighting coefficients are determined through rehabilitation movement analysis experiments. The weighting coefficients are set considering the importance of different parameters in representing movement abnormalities. The weighting coefficient for forward change emphasizes the contribution of instantaneous changes, the weighting coefficient for change consistency reflects the influence of overall sequence variation, and the weighting coefficient for local change highlights the role of local mean shift. The output of the change intensity index calculation module is a numerical index corresponding to each change point. The index value is dimensionless, and its magnitude is positively correlated with the significance of the change in movement characteristics.

[0034] The sliding window mechanism of the change point identification module includes special boundary condition handling. When the window approaches the start and end points of the sequence, a mirror-fill method is used to expand the sequence, replicating the boundary values ​​as virtual data points. The local extremum detection algorithm adds an amplitude threshold constraint; only when the feature value change exceeds a preset threshold is it confirmed as a valid change point, avoiding misidentification of small fluctuations caused by noise. The amplitude threshold is adaptively adjusted based on the historical statistical values ​​of the action feature sequence, and the threshold update rule is dynamically calculated based on the standard deviation of the feature values ​​within the sliding window. The change point identification module maintains an independent state machine for each action feature sequence. The state machine records the current window position and extremum detection state, and state transitions are driven by the arrival of new data points. The time series analysis algorithm can be equipped with various extremum detection strategies, including detection methods based on the zero-crossing points of the derivative and detection methods based on curvature extrema. Strategy selection is set through configuration parameters. The change point identification module provides change point confidence output. The confidence score is calculated based on the signal-to-noise ratio of the feature value at the change point to the local background value. High-confidence change points are preferentially passed to downstream modules.

[0035] The change intensity index calculation module's preset time window supports asymmetric settings; the window width before and after the change point can be configured independently to adapt to the asymmetry before and after the action change. Local change calculation introduces a weighted average algorithm, assigning greater weight to feature values ​​closer to the change point. The weighted average uses a Gaussian kernel function to generate the weight distribution. Change consistency calculation incorporates robustness processing; the maximum value of the first-order difference is selected to exclude the influence of obvious outliers, and outlier identification is based on the statistical quartile range of the difference values. Predefined weighting functions support non-linear combination methods. The weighting function employs a multilayer perceptron neural network structure, which obtains a non-linear mapping relationship through training on historical action data. The neural network structure of the weighting function includes three neurons in the input layer corresponding to three input parameters, five neurons in the hidden layer using the Sigmoid activation function, and one neuron in the output layer outputting the change intensity index. The change intensity index calculation module provides a result caching mechanism; the most recently calculated index value is stored in a fast cache, and repeated queries for the same change point directly return the cached value. The change intensity index calculation module and the change point identification module communicate via a message queue. The change point identification module encapsulates newly detected change points into messages and places them in the queue. The change intensity index calculation module retrieves the messages from the queue to trigger the calculation. The message structure includes a change point timestamp, a feature sequence identifier, and a feature value. The message passing mechanism ensures decoupling and asynchronous processing between modules. The calculation results of the change intensity index calculation module are written to a distributed database. The database index is built based on patient identifiers and session timestamps, supporting historical data backtracking analysis.

[0036] Real-time performance monitoring of the change point identification module is achieved through timestamp difference calculation. An alarm is triggered when the difference between the input and output timestamps exceeds a threshold, and the alarm information is recorded in the system log. The sliding window size adjustment strategy is based on online estimation of action frequency. The action frequency is obtained by analyzing the power spectrum of the action feature sequence using Fast Fourier Transform, and the window size matches the period of the dominant frequency component. The change point identification module supports parallel processing of multi-dimensional action feature sequences. Each feature dimension independently performs change point detection, and the final change point list merges the detection results from all dimensions. The weight coefficients of the change intensity index calculation module support online learning and updating. The weight coefficients are dynamically adjusted based on the classification performance of the latest rehabilitation action samples, and the online learning uses the stochastic gradient descent algorithm. The forward change calculation introduces direction sensitivity, distinguishing different weights for positive and negative changes for a specific rehabilitation action. Directional information is determined by the peak and trough values ​​of the change point type. The change consistency calculation is extended to multi-scale analysis, calculating the maximum value of the first-order difference at different time scales. The multi-scale results are weighted and fused to generate a comprehensive change consistency value. The calculation of local changes incorporates a statistical significance test, using a T-test to assess the significance of differences in the mean values ​​of features before and after the change point. Local changes are only included when the differences are significant. The output values ​​of the change intensity index calculation module are standardized to between zero and one hundred using a min-max scaling method, with scaling parameters determined based on historical data distribution. The integration test of the change point identification module and the change intensity index calculation module uses a standard sine wave superimposed with a step signal as test cases. The test cases cover various change point types and change intensity patterns to verify the correctness of the module's functionality.

[0037] Example 3: The average change intensity index acquisition module is configured to check whether each time point is identified as a change point in all action feature sequences. When the condition is met, the mean of the change intensity indices of all action feature sequences at that time point is calculated as the average change intensity index. The time point alignment mechanism is based on the global timestamp of the multimodal visual data sequence. The global timestamp is distributed by the system master clock, and each data point of the action feature sequence carries a time stamp accurate to the millisecond level. The average change intensity index acquisition module maintains a time point status table, which records whether each time point is marked as a change point in each action feature sequence. The status table update is triggered by the output of the change point identification module. When the change point status bits of all action feature sequences corresponding to a certain time point are set, the average change intensity index acquisition module performs the aggregation calculation of the change intensity index. The mean of the change intensity index is calculated using the arithmetic mean method, which sums the change intensity index values ​​of all action feature sequences at that time point and then divides it by the total number of action feature sequences. The output of the average change intensity index acquisition module is a time series. The time series only contains the index values ​​corresponding to time points that meet the synchronous change condition; index values ​​for time points with asynchronous changes are recorded as invalid. The time point status table cleanup mechanism periodically removes expired time point records to prevent memory overflow. The cleanup cycle is dynamically adjusted according to the data acquisition frequency.

[0038] The motion error index calculation module uses a clustering algorithm to cluster the average change intensity indices of all visual sensors, obtaining clusters. The clustering algorithm uses the K-means clustering method. The input data for the K-means clustering algorithm is a feature vector composed of the average change intensity index values ​​of all visual sensors at the same time point, and the dimension of the feature vector is equal to the number of visual sensors. The number of clusters K is determined by the elbow rule, which calculates the sum of squared clustering errors corresponding to different K values, and selects the inflection point of the rate of decrease of the sum of squared errors as the optimal K value. The clustering process iteratively updates the cluster centroids until the distance moved by the centroids is less than the convergence threshold. The clustering results group the visual sensors into different clusters. For each visual sensor, the motion error index calculation module calculates the mean of the Euclidean distance between its local window sequence of motion feature sequence and the corresponding local window sequences of motion feature sequences of other visual sensors in the same cluster as the trend dissimilarity index. The local window sequence is a fixed-length subsequence centered on the current calculation time point. The window length is set based on half a cycle of the rehabilitation action, and the window boundary is processed using a symmetrical filling method. The Euclidean distance is calculated by taking the square root of the sum of the squares of the differences between corresponding points in two local window sequences; the distance value reflects the degree of difference in sequence morphology. The trend dissimilarity index is obtained by averaging the Euclidean distances of all other sensors within the same cluster; a larger index value indicates poorer consistency between the sensor's motion characteristics and those of similar sensors. The motion error index calculation module obtains the motion error index based on the average change intensity index and the trend dissimilarity index through a proportional function. The proportional function is designed to consider the relative contributions of the two indices. Motion Error Index The calculation formula is: ; in: This represents the motion error index of the visual sensor at time point t. This represents the average change intensity index of the visual sensor at time point t. This represents the trend dissimilarity index of visual sensors at time point t. This represents the standard deviation of the average change intensity index of all visual sensors at time point t. The exponential function in the formula amplifies the influence of the trend dissimilarity index, and the standard deviation term in the denominator balances the group fluctuation factor.

[0039] The average change intensity index acquisition module employs multi-threaded parallel processing for time-point synchronization detection. Each action feature sequence is assigned an independent monitoring thread, and threads are synchronized via semaphores. When all threads report a change point at a certain time, the main thread triggers mean calculation. This parallel architecture improves the processing efficiency of the multi-sensor system. Robustness is added to the mean calculation of the change intensity index, excluding outliers that significantly exceed reasonable ranges. Outlier identification is based on the statistical distribution characteristics of historical data. The time series of the average change intensity index is stored in a circular buffer, covering a complete rehabilitation action cycle, supporting real-time streaming processing. The time-point status table uses a hash table as its data structure, with timestamps as keys and boolean arrays as values. The boolean arrays record the change point status of each action feature sequence at that time. The K-means clustering algorithm in the action error index calculation module is initialized using the K-means method. The K-means method optimizes the selection of initial centroids, avoiding getting trapped in local optima. An empty cluster detection mechanism is added to the clustering process. When a cluster is empty, centroids are reassigned to ensure that all clusters contain at least one sensor. The trend dissimilarity index calculation optimizes the distance metric. Before Euclidean distance calculation, the local window sequence is z-score standardized to eliminate the influence of dimensions. The selection of the local window sequence considers the action phase alignment issue; the starting point of the window is fine-tuned based on cross-correlation analysis of action feature sequences to ensure that the compared sequence segments are in the same action phase. The parameters of the proportional function are calibrated using a rehabilitation action standard template containing correct action data from healthy subjects. The parameter adjustment aims to generate higher index values ​​for erroneous actions. The output of the action error index calculation module is a two-dimensional array, with the array dimension being the number of time points multiplied by the number of visual sensors. Each element corresponds to the index value of a specific time and sensor.

[0040] The data interface between the average change intensity index acquisition module and the change intensity index calculation module adopts a publish-subscribe model. The change intensity index calculation module publishes new index values, and the average change intensity index acquisition module subscribes to relevant topics. Clock synchronization of the data sequence uses a network time protocol; time deviations between sensor nodes and processing nodes are periodically corrected, and resynchronization is triggered when the deviation exceeds a threshold. The clustering analysis period of the movement error index calculation module is configurable, adjusted according to the stability of rehabilitation movements; extending the clustering interval during stable periods reduces computational load. The trend dissimilarity index calculation supports weighted Euclidean distance, assigning greater weight to recent data points in the distance calculation, with the weighting coefficient decaying exponentially over time. The nonlinear design of the proportional function enhances sensitivity to abnormal movements, the exponential term amplifies the impact of inconsistent sensors, and the standard deviation term suppresses misjudgments caused by group fluctuations. The real-time performance of the movement error index calculation module is controlled by computational complexity; an upper limit is set on the local window length to avoid excessive computational overhead, and clustering analysis is performed during system idle periods. Data persistence for the average change intensity index acquisition module uses a time-series database, with the database index built based on patient session identifiers, supporting long-term trend analysis. The error handling mechanism of the action error index calculation module detects invalid input data. When sensor data is lost, the last valid value or interpolation estimate is used to ensure continuous system operation.

[0041] The distance metric for clustering algorithms can be extended to Mahalanobis distance, which considers the covariance structure of the dataset and appropriately discounts highly correlated features. The trend dissimilarity index calculation introduces a dynamic time warping algorithm, which handles inconsistent sequence lengths and adapts to variable-speed action execution. The denominator of the proportional function can be replaced with the average distance within clusters, enhancing the relative comparison between similar sensors. The action error index calculation module provides confidence assessment of the index value, calculated based on the silhouette coefficient of the clustering results. The silhouette coefficient measures the cohesion and dissimilarity of the clusters. The time point alignment tolerance of the average change intensity index acquisition module is configurable, allowing data within small time deviations to be considered synchronous, with the tolerance threshold adaptively adjusted according to the action speed. The multimodal fusion of the action error index calculation module supports sensor reliability weighting, with reliability weights calculated based on the signal-to-noise ratio of historical sensor data, giving higher-reliability sensors a larger voting weight. The number of clusters K can be dynamically adjusted; when the sensor data distribution changes, the optimal K value is re-evaluated, and the clustering model is updated online to adapt to changes in action patterns. The local window used in the trend dissimilarity index calculation supports multi-scale analysis. Windows at different scales capture short-term jitter and long-term drift, and the multi-scale results are fused to generate a comprehensive dissimilarity measure. The output range of the proportional function is standardized to between zero and one. The standardization uses the sigmoid function to compress extreme values, facilitating unified processing by subsequent modules. Data transfer between the action error index calculation module and the visual dynamic coefficient calculation module adopts zero-copy shared memory, reducing the overhead of copying large amounts of data and improving system throughput.

[0042] Example 4: See Figure 3 The visual dynamic coefficient calculation module is configured to use the action error index of each visual sensor as input and employ a threshold segmentation algorithm to obtain the optimal segmentation threshold. The threshold segmentation algorithm uses the maximum inter-class variance (MOL) method. The MOL method analyzes the histogram distribution of the action error index values, calculates the inter-class variance after dividing the data into two classes using different candidate thresholds, and selects the candidate threshold corresponding to the maximum inter-class variance as the optimal segmentation threshold. Time points where the action error index is greater than or equal to the optimal segmentation threshold are marked as high-error times, and time points where the action error index is less than the optimal segmentation threshold are marked as low-error times. The visual dynamic coefficient calculation module calculates the absolute value of the difference between the mean action error index at high-error times and the mean action error index at low-error times as the behavior difference index, and calculates the ratio of the number of low-error times to the total number of time points as the stationary duration ratio. Based on the mean action error index at high-error times, the behavior difference index, and the stationary duration ratio, the visual dynamic coefficient calculation module obtains the visual dynamic coefficient through a product ratio operation. The product ratio operation multiplies the three factors and then divides by a normalization constant.

[0043] The skeletal pose coefficient calculation module is configured to extract pitch, roll, and yaw angles from the motion feature sequence at each time point for each visual sensor as pose angle information. Pitch angle describes the angle of forward and backward tilt of a joint, roll angle represents the angle of left and right roll, and yaw angle reflects the angle of horizontal rotation of a joint. The module calculates the average rate of change of pose angles between consecutive time points at high error moments as the skeletal pose coefficient. The rate of change of pose angles is calculated by the difference between angle values ​​at adjacent time points. The rate of change of angles is calculated independently for each angle type of each joint, and the average is used to summarize the rate of change information for all joints and all angle types. The threshold segmentation algorithm of the visual dynamic coefficient calculation module includes a preprocessing step. The motion error index sequence is first passed through a median filter to remove impulse noise, and the window width of the median filter is set to an odd number of data points. The search range for the optimal segmentation threshold is limited to the minimum and maximum values ​​of the motion error index. The search step size is set according to the data accuracy requirements, and the weighted average formula is used to calculate the inter-class variance. The labeling results for high and low error moments are stored as Boolean arrays, which correspond to the original time series indices for easy reference in subsequent calculations. The behavioral difference index is calculated using an arithmetic mean formula. The sum of the action error indices at high error times is divided by the number of high error times; the sum of the action error indices at low error times is divided by the number of low error times. The absolute value of the difference between the two means is then taken. The total number of time points in the stationary duration ratio calculation refers to the total number of sampling points within a complete analysis window. The analysis window length is set according to the duration of the rehabilitation training phase.

[0044] The angle extraction in the skeletal posture coefficient calculation module is based on the 3D coordinates of skeletal key points. Pitch angle is calculated by the angle between the forward and backward direction vectors of the joint, roll angle by the angle between the left and right direction vectors of the joint, and yaw angle by the angle between the up and down direction vectors of the joint. Angle calculations are performed using the vector dot product formula and the inverse cosine function, with the unit of angle uniformly set to degrees. The determination of high-error moments depends on the output of the visual dynamic coefficient calculation module, and the two modules share time point marker information through shared memory. The attitude angle change rate calculation uses the central difference method; the change rate at the current time point is obtained by the difference between the angle values ​​of the two preceding and following time points, and the boundary points use forward or backward difference. The average change rate calculation considers all valid change rate values ​​within all high-error moment intervals, and abnormal change rates exceeding the physiological range are excluded before arithmetic averaging. The product ratio operation mathematical expression of the visual dynamic coefficient calculation module is the product of three factors divided by one hundred, with the normalization constant fixed at one hundred to ensure that the visual dynamic coefficient falls within a reasonable numerical range. The visual dynamic coefficient calculation module outputs a scalar value, which is updated each time the analysis window slides. The sliding step size of the analysis window is less than the window length to achieve a smooth transition. The skeletal pose coefficient calculation module supports a weighted average mode for calculating the average rate of angle change. The weights are set based on the magnitude of the angle change, with larger changes assigned higher weights. The skeletal pose coefficient calculation module generates independent skeletal pose coefficient values ​​for each visual sensor. The coefficients from multiple sensors are fused into an overall skeletal pose coefficient through arithmetic or weighted average. Refer to Table 1 for the parameter configuration of the threshold segmentation algorithm.

[0045] Table 1: Threshold Segmentation Algorithm Parameter Configuration Parameter name default value describe Median filter window width 5 points Window size for denoising action error exponential sequences Threshold search step size 0.01 Precision step size for optimal segmentation threshold search Methods for calculating inter-class variance weighted average Variance calculation formula based on weighted average of two types of data Minimum effective data ratio 10% The minimum proportion of data points required for each category The visual dynamic coefficient calculation module incorporates a robustness check for the behavior difference index calculation. When the number of high-error or low-error moments is insufficient, an alternative calculation strategy is employed, using the global mean instead of the category mean. The total number of time points in the denominator of the stationary duration ratio excludes invalid data points, which are those where the sensor lost signal or the data quality was substandard. The input parameters for the product ratio calculation are standardized: the mean of the action error index at high-error moments is divided by the global maximum value of the action error index, and the behavior difference index is divided by the global range of the action error index. The stationary duration ratio itself is a proportional value and does not require additional standardization. The output values ​​of the visual dynamic coefficient calculation module undergo smoothing filtering, with the filtering window covering multiple consecutive analysis periods to reduce abrupt changes in coefficient values. The angle calculation in the skeletal posture coefficient calculation module uses quaternion interpolation to improve angle estimation accuracy; quaternion representation avoids the gimbal lock problem of Euler angles. The angle change rate calculation introduces a low-pass filter, with the filter cutoff frequency set according to the maximum physiological frequency of human movement to remove spurious changes caused by high-frequency noise. The calculation of the angle change rate in high-error intervals considers the difference in interval length. For longer intervals, the moving average of the change rate is calculated, while for shorter intervals, the slope of a linear fit is used as the change rate. The data interface between the skeletal posture coefficient calculation module and the feature sequence extraction module uses direct memory access to avoid data copying delays, allowing angle calculation and feature extraction to proceed in parallel. The threshold segmentation algorithm in the visual dynamic coefficient calculation module supports simultaneous processing of multi-dimensional action error indices. Multi-dimensional indices are first reduced in dimensionality through principal component analysis, and the first principal component after dimensionality reduction is used as the threshold segmentation input. The behavioral difference index calculation is extended to multi-scale analysis. Behavioral difference indices at different time scales are weighted and fused to generate a comprehensive index, with the time scale based on the sub-stage division of rehabilitation actions. The calculation of the stationary duration ratio introduces time decay weights, assigning higher weights to recent time points, and the decay coefficient is calculated based on an exponential decay function. The product ratio operation in the visual dynamic coefficient calculation module allows adjustment of factor weights, with the weight coefficients adaptively optimized through a feedback loop based on rehabilitation training effects. The angle reference coordinate system of the skeletal posture coefficient calculation module is the world coordinate system, defined based on the initial posture calibration results, with the origin located at the center of the rehabilitation training area. The calculation order of pitch, roll, and yaw angles follows the ZYX Euler angle convention. The rotation order affects the numerical results of the angles but not the calculation of the rate of change. The unit of the rate of change of angles is uniformly set to degrees per second, consistent with the International System of Units (SI), making it easier for clinicians to understand the results. The output values ​​of the skeletal posture coefficient calculation module are subjected to outlier detection. An outlier threshold is set based on the statistical distribution of historical data, and coefficient values ​​exceeding the threshold are limited to a reasonable range. The collaborative work between the visual dynamic coefficient calculation module and the skeletal posture coefficient calculation module is achieved through an event-driven mechanism. When a new sequence of motion error indices arrives, the visual dynamic coefficient calculation is triggered, and the completion of the visual dynamic coefficient calculation triggers the skeletal posture coefficient calculation.The calculation results from both modules are encapsulated into data structures and written to a message queue. The neural network parameter adjustment module reads data from the queue for parameter updates. The historical data storage of the visual dynamic coefficient calculation module uses a circular buffer structure, with the buffer size storing data from the most recent complete training session, supporting real-time backtracking analysis. The angle calculation optimization of the skeletal pose coefficient calculation module uses a lookup table method to pre-calculate the sine and cosine values ​​corresponding to common angles, reducing the overhead of trigonometric function calls in real-time calculations.

[0046] See Figure 4 This chart presents the monitoring and analysis results of movement errors during pulmonary rehabilitation training. The blue curve represents the time series of the movement error index after median filtering, reflecting the quality of patient movement execution during training. Red scatter dots mark high-error moments, indicating significant deviations in patient movement execution; green scatter dots represent low-error moments, indicating relatively standard movement execution. The orange dashed line in the chart shows the optimal segmentation threshold calculated using the Otsu's method, which scientifically divides the entire training process into high-error and low-error states. The text boxes display key calculation results indicators, including important parameters such as the visual dynamics coefficient, behavioral difference index, and stationary duration ratio. These data collectively constitute a quantitative evaluation system for the quality of patient rehabilitation training. The concentrated distribution of high-error moments reflects the difficulties patients encounter in certain specific training phases, while the stationary duration ratio reflects the stability of the overall training process. The behavioral difference index quantifies the degree of difference between high and low error states, providing data support for the development of personalized rehabilitation programs.

[0047] Example 5: The neural network parameter adjustment module is configured to normalize the visual dynamic coefficient and skeletal posture coefficient. The normalized visual dynamic coefficient and skeletal posture coefficient are added together and multiplied by a preset initial parameter value to obtain the adjusted neural network parameter value, which is used to update the neural network weight matrix. Taking a specific pulmonary rehabilitation exercise as an example, the patient performs an upper limb extension training exercise. A multimodal visual sensor collects ten minutes of training data. The visual dynamic coefficient calculation module outputs a visual dynamic coefficient value of 0.85, and the skeletal posture coefficient calculation module outputs a skeletal posture coefficient value of 0.62. The normalization process of the neural network parameter adjustment module uses a minimum-maximum scaling method, with the scaling range set between zero and one. The normalization of the visual dynamic coefficient is based on the statistical characteristics of historical session data. The maximum recorded value of the historical visual dynamic coefficient is 1.20, and the minimum recorded value is 0.10. Substituting the current visual dynamic coefficient value of 0.85 into the normalization formula, the normalized visual dynamic coefficient value is calculated to be 0.75. The normalization of the skeletal pose coefficients was performed using a similar method. The historical maximum value of the skeletal pose coefficient was recorded as 1.50, and the minimum value as 0.05. The current skeletal pose coefficient value of 0.62 was normalized to obtain a normalized skeletal pose coefficient value of 0.41. The normalization process eliminates the dimensional differences between different coefficients, making the values ​​comparable.

[0048] The neural network parameter adjustment module adds the normalized visual dynamic coefficients and skeletal pose coefficients, and the summation operation yields a comprehensive adjustment factor of 1.16. Preset initial parameter values ​​are set according to the neural network structure and training objectives, and are stored in a configuration file; in this example, the preset initial parameter value is 0.50. The neural network parameter adjustment module performs a multiplication operation, multiplying the comprehensive adjustment factor value of 1.16 by the preset initial parameter value of 0.50 to obtain an adjusted neural network parameter value of 0.58. This adjusted value is applied to the update process of the neural network weight matrix. Each element of the weight matrix is ​​scaled proportionally, with the scaling ratio determined based on the adjusted value. The weight matrix update of the neural network parameter adjustment module uses element-wise multiplication. The original weight matrix W is multiplied by the adjusted value matrix A using a Hadamard product operation, and all elements of the adjusted value matrix A have a value of 0.58. The updated weight matrix W' is calculated using the formula W' = W⊙A, where ⊙ represents element-wise matrix multiplication. The weight matrix update operation is performed during the forward propagation of the neural network, and the update frequency is synchronized with the update cycle of the visual dynamic coefficients and skeletal pose coefficients. The neural network parameter tuning module maintains a parameter update log, recording the timestamp, coefficient value, and adjustment result for each adjustment. The historical data window for normalization is configurable; its size determines the update speed of the normalization parameters, and is set to the data from the most recent one hundred training sessions. Boundary handling in the min-max scaling method includes outlier detection; when coefficient values ​​exceed the historical range, a boundary value truncation strategy is employed to prevent distortion of the normalization results. The calculation of the comprehensive adjustment factor supports a weighted summation mode, where visual dynamic coefficients and skeletal pose coefficients can be assigned different weights, set based on clinical expert experience. The determination of preset initial parameter values ​​is optimized using neural network architecture search technology, which evaluates the impact of different parameter values ​​on action recognition accuracy on the validation set.

[0049] The weight matrix update process considers the characteristics of different layers in the neural network. Convolutional layers and fully connected layers can be configured with different adjustment strategies, and the adjustment values ​​for convolutional layers are weighted based on the importance of spatial features. The neural network parameter tuning module provides a rollback mechanism, restoring the previous effective parameter settings when parameter tuning leads to performance degradation. The rollback decision is based on the accuracy change on the validation set. The adjustment value application supports a partial update mode, updating only the parameters of specific layers in the neural network related to action recognition, preserving the stability of the underlying feature extraction layers. The parameter update log data structure includes session identifier, patient number, and action type fields, supporting multi-dimensional queries and analysis. The integration of the neural network parameter tuning module with the neural network inference engine is achieved through a dynamic link library. The function interface defines the input and output specifications for parameter tuning, and the interface is compatible with various deep learning frameworks. Historical data of normalized parameters is stored in a time-series database, which supports fast retrieval and aggregation queries, and a data backup mechanism prevents the loss of historical records. The optimization of the preset initial parameter values ​​uses a Bayesian optimization method. Bayesian optimization performs an intelligent search in the parameter space to find the optimal parameter values ​​that minimize the loss on the validation set. The parallel computation of weight matrix updates utilizes the multi-core architecture of the graphics processing unit (GPU), decomposing large-scale matrix multiplication operations into multiple parallel tasks to improve computational efficiency. The real-time performance of the neural network parameter adjustment module is guaranteed through an asynchronous processing mechanism; parameter adjustment calculations are executed in a background thread, without affecting the main forward inference thread of the neural network. Normalized parameter updates use a sliding window model, automatically updating the maximum and minimum value statistics when new data arrives and correcting the statistics when old data leaves the window. The calculation of the comprehensive adjustment factor introduces a nonlinear transformation function, which enhances the influence of larger coefficient values ​​and uses an S-curve for smoothing. Adaptive adjustment of preset initial parameter values ​​is based on network gradient information; the gradient magnitude reflects the sensitivity of parameter updates, with parameters of higher sensitivity assigned smaller initial values.

[0050] The weight matrix update impact assessment module monitors the effect of parameter changes on network output. Impact assessment is achieved by calculating the rate of change of the output feature map; if the rate of change exceeds a threshold, adjustment value correction is triggered. The neural network parameter adjustment module supports batch processing mode, simultaneously adjusting the parameters of multiple related neural networks, with parameter adjustment strategies shared across networks. The robustness of normalization processing is enhanced by adding a regularization term, which prevents drastic changes in the normalization result when coefficient values ​​fluctuate near the boundary. The range limitation of the comprehensive adjustment factor is achieved through a shearing function, which restricts factor values ​​within a reasonable range, avoiding extreme values ​​that could lead to uncontrolled network parameters. The hierarchical setting of preset initial parameter values ​​adapts to the depth structure of the neural network; shallow parameters use larger initial values ​​to retain low-level features, while deep parameters use smaller initial values ​​to focus on high-level semantics. The sparsity processing of weight matrix updates only updates parameter elements that have changed significantly. Sparsity is based on a threshold for parameter change magnitude filtering, reducing computational and storage overhead. The verification process of the neural network parameter adjustment module includes an integrity check, verifying whether parameter values ​​are within a physically reasonable range; abnormal parameter values ​​trigger alarms and manual intervention. The weighting strategy for normalized historical data emphasizes the importance of recent data, with the weights decaying exponentially over time, gradually reducing the influence of older data. In the example of the neural network parameter adjustment module, when a patient performs upper limb extension training movements, a visual dynamic coefficient of 0.85 reflects significant dynamic errors during movement execution, while a skeletal posture coefficient of 0.62 indicates a rapid rate of change in joint angles. Normalization maps the original coefficients to a standard range, eliminating dimensional differences between different rehabilitation movement types. A comprehensive adjustment factor of 1.16, greater than one, indicates a need to enhance the neural network's sensitivity to abnormal movements. After applying an adjustment value of 0.58 to the weight matrix, the neural network's ability to detect error patterns in similar upper limb extension training movements is improved. The parameter adjustment process is automatically executed three times during a ten-minute training period, with each adjustment based on the latest calculated visual dynamic coefficient and skeletal posture coefficient, achieving dynamic optimization of the neural network parameters. The implementation of the neural network parameter adjustment module ensures that the pulmonary rehabilitation movement correction system can adapt to the movement characteristics of different patients, optimizing movement recognition and correction performance through real-time adjustment of neural network parameters. Normalization ensures the comparability of coefficient values ​​from different sensors, while comprehensive adjustment factors integrate multi-dimensional information, and preset initial parameter values ​​provide a basic adjustment scale. The weight matrix update mechanism ensures that the neural network maintains its sensitivity to abnormal movements, improving the safety and effectiveness of pulmonary rehabilitation training.

[0051] See Figure 5This diagram illustrates the adaptive adjustment process of neural network parameters during training. The four curves in the figure represent the normalized visual dynamic coefficient, the normalized skeletal posture coefficient, the comprehensive adjustment factor, and the final adjusted neural network parameter value, respectively. The curve changes reflect the process by which the system dynamically adjusts the neural network parameters based on the patient's real-time training performance. The normalized visual dynamic coefficient curve shows the standardized results output by the visual analysis module, while the normalized skeletal posture coefficient reflects the quantitative assessment of skeletal movement posture. The comprehensive adjustment factor integrates these two important indicators, forming a comprehensive guide for neural network adjustment. The changing trend of the neural network parameter adjustment value reflects the system's response mechanism to the patient's training status. When an increase in movement errors or abnormal posture is detected, the system adjusts the neural network parameters accordingly to enhance the ability to identify abnormal patterns. This adaptive mechanism ensures that the rehabilitation training system can be optimized in real time for individual differences and training progress of different patients, improving training effectiveness and safety.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multimodal visual neural network-guided lung rehabilitation movement correction system, characterized in that, include: The visual data acquisition module is configured to acquire multimodal visual data sequences of patients' pulmonary rehabilitation movements; The feature sequence extraction module is configured to extract action feature sequences based on multimodal visual data sequences. The change point recognition module is configured to identify change points from action feature sequences; The change intensity index calculation module is configured to obtain the change intensity index based on the feature change magnitude at the change point, the variation of the feature sequence, and the mean change of the local feature sequence. The module for obtaining the average change intensity index is configured to obtain the average change intensity index based on the change intensity index. The motion error index calculation module is configured to obtain the motion error index based on the distance between motion feature sequences and the average change intensity index. The visual dynamic coefficient calculation module is configured to obtain visual dynamic coefficients based on the action error index. The skeleton pose coefficient calculation module is configured to obtain skeleton pose coefficients based on pose angle information in action feature sequences. The neural network parameter adjustment module configures the parameters of the neural network based on visual dynamic coefficients and skeletal pose coefficients.

2. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 1, characterized in that, The visual data acquisition module includes a multimodal visual sensor configured to simultaneously acquire RGB image sequences, depth image sequences, and infrared image sequences of the patient, wherein the multimodal visual sensor is arranged around the patient to cover the full range of rehabilitation movements.

3. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 2, characterized in that, The feature sequence extraction module is configured to extract skeletal key point sequences as motion feature sequences from RGB image sequences, depth image sequences, and infrared image sequences, wherein the skeletal key point sequences include joint positions and motion trajectories.

4. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 3, characterized in that, The change point identification module is configured to use a time series analysis algorithm to detect peaks and valleys in each action feature sequence as change points, wherein the time series analysis algorithm calculates the local extrema of feature values ​​based on a sliding window.

5. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 4, characterized in that, The change intensity index calculation module is configured to, for each change point, calculate the absolute value of the difference between the feature value at the change point and the feature value at the previous moment as the forward change amount, obtain the change consistency amount based on the maximum value of the first-order difference of the action feature sequence, and take a preset time window centered on the change point to calculate the absolute value of the difference between the mean of features before the change point and the mean of features after the change point as the local change amount; and obtain the change intensity index based on the forward change amount, the change consistency amount, and the local change amount through a predefined weighting function.

6. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 5, characterized in that, The average change intensity index acquisition module is configured to, for each time point, when the time point is identified as a change point in all action feature sequences, calculate the mean of the change intensity indices of all action feature sequences at that time point as the average change intensity index.

7. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 6, characterized in that, The motion error index calculation module is configured to use a clustering algorithm to cluster the average change intensity index of all visual sensors to obtain clusters. For each visual sensor, the mean of the Euclidean distance between the local window sequence of its motion feature sequence and the local window sequence of the corresponding motion feature sequence of other visual sensors in the same cluster is calculated as the trend dissimilarity index. The motion error index is obtained by using a scaling function based on the average change intensity index and the trend dissimilarity index.

8. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 7, characterized in that, The visual dynamic coefficient calculation module is configured to use the action error index of each visual sensor as input and a threshold segmentation algorithm to obtain the optimal segmentation threshold. Time points where the action error index is greater than or equal to the optimal segmentation threshold are marked as high error times, and time points where the action error index is less than the optimal segmentation threshold are marked as low error times. The absolute value of the difference between the mean action error index at high error times and the mean action error index at low error times is calculated as the behavior difference index. The ratio of the number of low error times to the total number of time points is calculated as the stationary duration ratio. Then, the visual dynamic coefficient is obtained by multiplying the mean action error index at high error times, the behavior difference index, and the stationary duration ratio.

9. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 8, characterized in that, The skeletal attitude coefficient calculation module is configured to extract the pitch angle, roll angle and yaw angle at each time point from the action feature sequence for each visual sensor as attitude angle information, and calculate the average value of the attitude angle change rate between consecutive time points under high error time as the skeletal attitude coefficient.

10. The multimodal visual neural network-guided lung rehabilitation movement correction system as described in claim 9, characterized in that, The neural network parameter adjustment module is configured to normalize the visual dynamic coefficients and skeletal pose coefficients, add the normalized visual dynamic coefficients and skeletal pose coefficients together and multiply them by a preset initial parameter value to obtain the adjustment value of the neural network parameters, which is used to update the weight matrix of the neural network.

Images