A Virtual Reality Rehabilitation Training Method and System Based on Motion Comparison Feedback

Abstract

This invention relates to the field of healthcare informatics technology, specifically to a virtual reality rehabilitation training method and system based on motion contrast feedback. The method includes: generating standardized relative motion data; calculating the motion completion score, motion fluency score, and coordination score for each key point based on the standardized relative motion data, and weighting and fusing them to obtain the current anomaly index for each key point; calculating the structural centrality weight and mechanical leverage weight for each key point, and weighting and fusing them to obtain the next-stage importance weight for each key point; using the current anomaly index and the next-stage importance weight, obtaining the next-stage limitation index for each key point; calculating the predicted completion rate based on the next-stage limitation index, and then calculating the total stage fit score; and determining whether the target subject is allowed to enter the next stage of training based on the total stage fit score. This invention can improve the adaptability and risk control capabilities of rehabilitation training.

Description

Technical Field

[0001] This invention relates to the field of healthcare informatics technology, specifically to a virtual reality rehabilitation training method and system based on motion contrast feedback. Background Technology

[0002] Currently, virtual reality technology is increasingly widely used in the field of rehabilitation training. Most existing VR rehabilitation training systems adopt a linear, progressive model. That is, the system sets the training content and assessment standards for each preset rehabilitation stage. Once the patient meets the assessment requirements of the current stage, the system automatically unlocks the training content for the next stage. This linear progression mechanism has certain limitations in clinical practice.

[0003] The human musculoskeletal system is a highly coordinated whole, with complex biomechanical couplings between joints and muscle groups—the holistic interconnectedness of the "human kinetic chain." Current progressive training models often overlook this characteristic, focusing only on whether the patient completes the prescribed movements at the current stage, without fully considering the quality of those movements and their potential impact on subsequent stages. As training progresses to higher levels, the complexity and load of movements increase significantly, potentially triggering new biomechanical changes. For example, in complex shoulder joint movements, if a patient develops abnormal compensatory mechanisms (such as shoulder shrugs or trunk rotations) due to insufficient muscle strength or lack of control, while they may temporarily achieve the movement goal, this compensatory pattern may negatively interfere with the elbow and wrist joints that have stabilized in the previous stage, even leading to a decline in their functional indicators. This risk of "paying for one thing while neglecting another," if not systematically monitored and intervened in real time, will seriously affect the overall rehabilitation outcome and even increase the risk of secondary injury.

[0004] Therefore, how to not only assess the patient's current performance during rehabilitation training, but also predict the impact of their movement patterns on the safety and effectiveness of the next stage of training, so as to ensure the steady progress of rehabilitation while avoiding functional regression caused by inappropriate advancement, has become a technical problem that current VR rehabilitation training systems urgently need to solve. Summary of the Invention

[0005] This invention provides a virtual reality rehabilitation training method and system based on motion contrast feedback to solve existing problems.

[0006] The present invention provides a virtual reality rehabilitation training method and system based on motion comparison feedback, which adopts the following technical solution: One embodiment of the present invention provides a virtual reality rehabilitation training method based on motion contrast feedback, the method comprising the following steps: The original spatiotemporal sequence data of key points is acquired, and standardized relative motion data is generated based on the original spatiotemporal sequence data of key points. The original spatiotemporal sequence data of key points includes timestamps, node IDs, three-dimensional spatial coordinates and quaternion poses. The node IDs correspond to 135 preset human body key points. Based on standardized relative motion data, the action completion score, action fluency score, and coordination score of each key point when the target object performs the current action are calculated, and the weighted fusion is used to obtain the current anomalous index of each key point. Based on the standard actions for the next stage, calculate the structural centrality weight and mechanical lever weight of each key point, and then weight and fuse them to obtain the importance weight of each key point for the next stage. By using the current anomalous index of each key point and the importance weight of the next stage, the limitation index of each key point in the next stage is obtained. Based on the next-stage limitation index of each key point, the predicted completion rate of each action in the next stage is calculated, and then the overall stage adaptability score is calculated. The decision on whether to allow the target subject to proceed to the next stage of training is based on the overall stage fit score.

[0007] Furthermore, the specific steps for obtaining the original spatiotemporal sequence data of key points and generating standardized relative motion data based on the original spatiotemporal sequence data are as follows: Acquire the original RGB-D video frame sequence and the original signal from the IMU inertial sensor of the target object; The original RGB-D video frame sequence and the original signal from the IMU inertial sensor are input into a pre-trained 3D human pose estimation neural network to obtain the original spatiotemporal sequence data of key points. The three-dimensional spatial coordinates of each key point in the original key point spatiotemporal sequence data are transformed into relative coordinates relative to the shoulder joint; Based on the transformed coordinates of each key point, obtain the length of each limb vector; Obtain the standard length of each limb of the target object measured in the initial static posture, divide the length of each limb vector by its corresponding standard length, and obtain the normalized relative position vector of each key point. The three-dimensional spatial coordinates in the original keypoint spatiotemporal sequence data are replaced with the normalized relative position vectors of each keypoint to generate standardized relative motion data.

[0008] Furthermore, the specific steps for calculating the action completion score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Extract the maximum range of motion of the joints at each key point when the target object performs the current action from the standardized relative motion data, and record it as the actual peak angle of each key point. Obtain the ideal peak angles of key points of the target object when it performs the current action; For each key point, calculate the absolute value of the difference between its actual peak angle and the ideal peak angle. Divide the absolute value of the difference by the ideal peak angle to obtain the relative deviation ratio. Subtract the relative deviation ratio from 1 to obtain the action completion score for that key point.

[0009] Furthermore, the specific steps for calculating the motion smoothness score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Extract the instantaneous velocity of each key point in each frame when the target object performs the current action from the standardized relative motion data; For two adjacent frames, subtract the instantaneous velocity of the corresponding key point in the previous frame from the instantaneous velocity of each key point in the later frame, and then divide by the time difference between frames to obtain the acceleration of each key point in the later frame. The acceleration of each key point in the previous frame is obtained, the acceleration in the next frame is subtracted from the acceleration in the previous frame, and then divided by the time difference between frames to obtain the jerk of each key point in the next frame, forming an acceleration sequence. A sliding window average is applied to the jerk sequence, and the average jerk within the window is used as the smoothed jerk of each key point in that frame. Calculate the value of an exponential function with the natural constant e as the base and the product of a negative attenuation coefficient and smoothed jerk as the exponent to obtain the instantaneous smoothness score of the key point in each frame; take the average of the instantaneous smoothness scores of all frames in the entire motion process as the motion smoothness score of the key point.

[0010] Furthermore, the specific steps for calculating the coordination score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Extract active key point displacement trajectory data and compensatory key point displacement trajectory data of the target object when performing the current action from standardized relative motion data; For each active key point, identify at least one corresponding compensatory key point; Calculate the cumulative displacement of each compensation key point of the target object from the start of the current action to the end of the current action; Divide the cumulative displacement by the preset compensation tolerance threshold to obtain the violation rate of the compensation key point. Subtracting the proportion of violations from 1 yields the synergy score for this key compensation point. If the current active key point involves multiple compensatory key points, then the minimum value of the synergy score of all compensatory key points is taken as the synergy score of the active key point. The weighted fusion process yields the current anomalous index for each key point, and includes the following specific steps: The weighted sum of the action completion score, action fluency score, and coordination score of each key point is calculated to obtain the positive comprehensive score of each key point; 1 is subtracted from the positive comprehensive score to obtain the current anomaly index of each key point.

[0011] Furthermore, the specific steps for calculating the structural centrality weights of each key point based on the standard actions of the next stage are as follows: Obtain the key point connection matrix for the target object to perform the next stage standard action; where the rows and columns of the matrix correspond to key points, a matrix element of 1 indicates that there is a direct connection between two key points, and a matrix element of 0 indicates that there is no direct connection between two key points; Degree centrality of each keypoint in the keypoint connection matrix; where degree centrality is the ratio of the number of keypoints directly connected to each keypoint in the matrix to the maximum possible number of connections. Find the shortest path for all keypoint pairs in the keypoint connection matrix; For each key point, calculate the proportion of the number of times the key point is visited in all shortest paths to the total number of shortest paths, and use the proportion as the betweenness centrality of the key point. The structural centrality weights of each key point are obtained by weighted summation of the degree centrality and betweenness centrality.

[0012] Furthermore, the specific steps for calculating the mechanical lever weights at each key point based on the standard action of the next stage are as follows: During the execution of the next stage of standard actions by the target object, obtain the maximum linear displacement or maximum spatial volume swept by the remote key points corresponding to each key point; Divide the maximum linear displacement or maximum spatial volume swept by the remote key point corresponding to each key point by the maximum value of that value among all key points to obtain the lever ratio of each remote key point. If a key point corresponds to multiple remote key points, then the maximum value among the leverage ratios of all remote key points is taken as the mechanical leverage weight of that key point.

[0013] Furthermore, the specific steps for obtaining the next-stage limitation index of each key point by utilizing the current anomaly index and the importance weight of the next stage are as follows: Multiply the current anomalous index of each key point by the importance weight of the key point in the next stage to obtain the limitation index of each key point in the next stage under a single action. If the next stage contains multiple actions, the current anomalousness index of each key point in each action is multiplied by the importance weight of that key point in the next stage under that action, and then summed and divided by the number of actions to obtain the limitation index of each key point in the next stage.

[0014] Furthermore, the specific steps for calculating the predicted completion rate of each action in the next stage based on the limitation index of each key point, and then calculating the overall stage adaptability score, are as follows: For each action in the next stage, obtain the maximum value of the next stage limitation index of all key points involved in the action, and subtract the maximum value from 1 to obtain the predicted completion rate of the action. The number of actions with a predicted completion rate greater than or equal to the preset completion rate threshold among all actions in the next stage is counted as the number of qualified actions; Divide the number of qualified actions by the total number of actions in the next stage to obtain the pass rate; The predicted completion rate of each action in the next stage is multiplied by the difficulty weight of the corresponding action, and the weighted average is calculated as the expected performance score. Obtain the preset first fusion coefficient and second fusion coefficient, multiply the pass rate by the first fusion coefficient, multiply the expected performance score by the second fusion coefficient, sum them, and then multiply by 100 to obtain the stage adaptability total score.

[0015] This invention proposes a virtual reality rehabilitation training system based on motion contrast feedback, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the virtual reality rehabilitation training method based on motion contrast feedback.

[0016] The beneficial effects of the technical solution of this invention are as follows: This invention proposes a virtual reality rehabilitation training method and system based on action comparison feedback. It not only evaluates the immediate rehabilitation effect of the current stage based on multiple dimensions such as action completion, fluency, and synergy, but also quantifies and predicts the potential impact of the current joint abnormality on the target joint in the next stage through algorithms such as topological centrality and mechanical leverage amplitude. The new stage is only unlocked when the prediction results show that the patient has the physiological basis for safe advancement; otherwise, the advancement is automatically locked and forced consolidation training is carried out. This effectively avoids the negative interference of abnormal compensation induced by high-order movements on the recovered joints, prevents functional regression, and ensures the integrity and safety of the rehabilitation process. At the same time, through 135-point whole-body model monitoring and individualized normalization processing, fair and accurate assessment from fine finger movements to large joint compound movements is achieved, which significantly improves the adaptability and risk control capabilities of rehabilitation training. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the steps of a virtual reality rehabilitation training method based on motion contrast feedback according to the present invention. Figure 2 This is a block diagram of a virtual reality rehabilitation training system based on motion contrast feedback according to the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a virtual reality rehabilitation training method and system based on motion contrast feedback proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] The following description, in conjunction with the accompanying drawings, details a specific scheme for a virtual reality rehabilitation training method and system based on motion contrast feedback provided by this invention.

[0022] For breast cancer patients, postoperative complications can include joint stiffness, lymphedema, pain, and muscle weakness. Joint stiffness: Post-operative joint stiffness in breast cancer patients can be caused by reduced activity and limited movement due to pain. This can be improved through massage and exercise. Lymphedema: Obstructed lymphatic drainage in the affected area can cause lymphedema, leading to swelling, pain, and limited movement in the affected upper limb. Lymphatic drainage can be promoted through massage and elevation of the affected limb. Rehabilitation training is often necessary to assist patients in recovering as quickly as possible.

[0023] The existing phased rehabilitation training is implemented as follows: Based on the basic principles of postoperative rehabilitation for breast cancer patients and their actual recovery status, patients are guided through phased rehabilitation training, with clear goals for each phase. Phase 1 (within 7 days post-surgery) focuses on functional training of the elbow, wrist, and metacarpophalangeal joints, aiming to reduce the incidence of lymphedema and promote lymphatic and blood return. Phase 2 (8-14 days post-surgery) focuses on shoulder joint movement, including shoulder shrugs, arm rotations, and pendulum exercises, aiming to prevent upper limb dysfunction. Phase 3 (15 days post-surgery) combines active limb movement, aerobic exercise, and resistance training, strictly adhering to a gradual progression principle, aiming to achieve free movement of the shoulder and elbow joints. Correspondingly, a virtual reality system will also provide phased training and recovery for various body parts of the patient.

[0024] Please see Figure 1 The diagram illustrates a flowchart of a virtual reality rehabilitation training method based on motion contrast feedback according to an embodiment of the present invention. The method includes the following steps: Step S001: Obtain the original spatiotemporal sequence data of key points, and generate standardized relative motion data based on the original spatiotemporal sequence data of key points; wherein, the original spatiotemporal sequence data of key points includes timestamps, node IDs, three-dimensional spatial coordinates and quaternion poses, and the node IDs correspond to 135 preset human body key points.

[0025] It should be noted that the system guides the patient step-by-step through a virtual wizard interface to properly wear the head-mounted display (HMD) and hand controllers, prompting them to adjust the tightness of the straps to ensure that the devices do not slip relative to each other during movement. Then, the "static coupling verification" procedure is initiated, requiring the patient to remain upright and still for 5 seconds. The system monitors the acceleration and angular velocity readings of the device's built-in IMU (Inertial Measurement Unit) in real time. If abnormal high-frequency shaking (indicating looseness) or a non-zero gravity vector (indicating tilt) is detected, visual feedback is immediately triggered to prompt the patient to readjust. This step aims to ensure a rigid connection between the sensor coordinate system and the human body segments from a physical perspective, eliminating subsequent data distortion caused by device slippage and laying a physical foundation for acquiring clean signals.

[0026] Step S001 further includes steps S0011-S0016: Step S0011: Obtain the original RGB-D video frame sequence of the target object and the original signal from the IMU inertial sensor.

[0027] It should be noted that the target group refers to patients who need rehabilitation training.

[0028] The original RGB-D video frame sequence of the target object refers to a time-series data stream composed of color images and depth images continuously acquired by the head-mounted display or depth camera. Each color image records the texture information of the scene, while the depth image provides the physical distance from each pixel to the camera. The combination of the two can be used for accurate calculation of subsequent three-dimensional spatial coordinates. The original signal from the IMU inertial sensor refers to the unfiltered or unfused acceleration and angular velocity data directly output by the accelerometer and gyroscope inside the head-mounted display and controllers, which retains the original dynamic characteristics such as high-frequency vibrations and instantaneous impacts during the movement.

[0029] Subsequently, the system simultaneously activates multi-source data acquisition channels at a high sampling rate of 90Hz: on one hand, it continuously acquires raw RGB-D video frame sequences, including color and depth image information, using the RGB-D camera built into the headset; on the other hand, it acquires raw acceleration and angular velocity signals from the IMU inertial sensors inside the headset and controllers in parallel. All data streams are accompanied by microsecond-level timestamps and directly written to a circular buffer, forming a raw data stream without any geometric calculations, filtering, smoothing, or scaling. This acquisition method preserves the original physical dimensions and temporal characteristics of the sensors.

[0030] Step S0012: Input the original RGB-D video frame sequence and the original signal from the IMU inertial sensor into the pre-trained 3D human pose estimation neural network to obtain the original spatiotemporal sequence data of key points.

[0031] It's important to note that, compared to common key point models: 14 / 17 / 18-point models primarily cover the main body and large joints of the limbs (shoulder, elbow, wrist, knee, etc.), but they typically don't include fine points for the fingers (metacarpophalangeal joints). 25-point models may be more detailed, but they also usually don't cover the fingers. Only OpenPose's 135-point "full-body" model includes points for the face, torso, limbs, and fine details for the hands (21 points per hand) and feet. This perfectly matches the metacarpophalangeal joint activities that need to be monitored in rehabilitation training.

[0032] Therefore, in order to comprehensively monitor the entire rehabilitation process from fine finger movements to large-range shoulder joint movements, this embodiment uses a 135-point model that includes fine finger points. It can meet the assessment of the fingers in the first stage and also cover the monitoring of shoulder and elbow movements in the latter two stages.

[0033] The first step in rehabilitation training is to "see" the patient. The system must act like the therapist's eyes, accurately capturing key bone points on the patient's body. For post-mastectomy breast cancer patients, it's crucial not only to focus on active force points such as the affected shoulder, elbow, and wrist, but also to monitor compensatory points like the cervical spine and pelvis, as patients often mask insufficient shoulder mobility by shrugging or twisting their waist. Only by converting these physical locations into digital coordinates can subsequent analysis become possible.

[0034] Specifically, the system adopts a 3D human pose estimation neural network based on the improved OpenPose architecture, which uses the original RGB-D video frame sequence and the original signal from the IMU inertial sensor as multimodal inputs.

[0035] The network employs an encoder-decoder structure. First, a convolutional neural network extracts a 2D heatmap of human key points from RGB-D frames. This heatmap, combined with a depth map, maps the 2D key points to 3D space. Simultaneously, an IMU signal is used to correct and smooth the motion trajectories of the key points through a temporal convolutional network. Finally, the network outputs the 3D spatial coordinates (x, y, z) and quaternion poses (qx, qy, qz, qw) of 135 preset human key points in each frame. These 135 key points specifically include: 25 points on the body and feet (including shoulder, elbow, wrist, hip, knee, ankle, and foot key points), 21 points on each hand (covering three joints and fingertips of each finger), and 70 points on the face (covering eyebrows, eyes, nose, lips, and facial contours).

[0036] The network output data is organized in frames. Each frame contains a timestamp, node IDs of 135 key points and their corresponding 3D coordinates and quaternion poses, forming the original spatiotemporal sequence data of key points.

[0037] Taking the right shoulder abduction movement as an example, the original spatiotemporal sequence data of key points is represented as a set of multiple frames arranged continuously in time. Each frame contains a microsecond-level timestamp and complete information on 135 human body key points. Taking the three key points of right shoulder (ID=10), right elbow (ID=20), and right wrist (ID=30) as examples, at the beginning frame of the movement (t=0.000s), the coordinates of the right shoulder are (0.20, 1.60, 0.10) meters and the quaternion posture is (0, 0, 0, 1), the coordinates of the right elbow are (0.55, 1.55, 0.12) meters, and the coordinates of the right wrist are (0.90, 1.60, 0.10) meters; up to the... At frame 3 (t=0.067s), as the arm rises, the coordinates of the right wrist change to (0.78, 1.65, 0.32) meters, the coordinates of the right elbow change to (0.48, 1.60, 0.25) meters, and the coordinates of the right shoulder remain relatively stable. Simultaneously, the quaternion poses of the right elbow and right wrist change to (0.02, 0, 0, 0.98) and (0.04, 0, 0, 0.96) respectively, reflecting the rotation process of the joints. The coordinates and pose data of all keypoints are continuously recorded at the original sampling frequency.

[0038] Through these operations, a patient's "digital skeleton" is constructed, mapping real-world movements into discrete mathematical points that can be processed by a computer, thus enabling subsequent calculations.

[0039] Step S0013: Transform the three-dimensional spatial coordinates of each key point in the original key point spatiotemporal sequence data into relative coordinates relative to the shoulder joint.

[0040] It's important to note that patients' height and arm length vary greatly. Taller patients naturally have a larger range of arm swing, while shorter patients have a smaller one. Directly comparing absolute distances can lead to misjudgments of shorter patients, who may be perceived as having incomplete movements. Furthermore, rehabilitation focuses on the angle and relative trajectory of joint rotation, not absolute spatial position. Therefore, the influence of height and arm length must be eliminated so that all patients are assessed under the same "relative standard."

[0041] Specifically, for each frame of data, the coordinates of the affected shoulder joint are selected as the origin of the local coordinate system. The global coordinates of all key points in the frame are subtracted from the shoulder joint coordinates to obtain the relative coordinates of each key point with respect to the shoulder joint. For example, if the shoulder joint coordinates in a certain frame are (0.20, 1.60, 0.10) meters and the wrist joint coordinates are (0.90, 1.60, 0.10) meters, then after translation, the relative coordinates of the wrist joint become (0.70, 0, 0) meters, which is the spatial vector from the shoulder joint to the wrist joint.

[0042] Step S0014: Based on the transformed coordinates of each key point, obtain the length of each limb vector.

[0043] It should be noted that, based on the transformed relative coordinates of each keypoint, the system calculates the vector length of each limb segment sequentially according to the predefined limb connection relationships of the human skeletal topology. Specifically, for each limb segment (such as the upper arm, forearm, thigh, and lower leg), the system identifies the node IDs of its proximal and distal keypoints, extracts the transformed relative coordinates of these two keypoints in each frame, and calculates the Euclidean distance between the two points, thus obtaining the actual length of the limb vector in the current frame. For example, the upper arm limb vector points from the shoulder joint to the elbow joint, and its length is the spatial straight-line distance between the relative coordinates of the shoulder and elbow points; the forearm limb vector points from the elbow joint to the wrist joint, and its length is the distance between the elbow and wrist points. By traversing all limb segments, the system obtains the temporal sequence of the length of each limb vector frame by frame, providing basic data for subsequent scaling transformations to eliminate individual body shape differences.

[0044] Step S0015: Obtain the standard length of each limb of the target object measured in the initial static posture, divide the length of each limb vector by its corresponding standard length, and obtain the normalized relative position vector of each key point.

[0045] It should be noted that, in the initial static standing posture, the system collects key point data of the target object while it remains upright and still through a static coupling verification process. Based on the relative coordinates of each key point at this time, the system calculates the initial vector length of each limb segment, which is then used as the standard length of the corresponding limb of the target object. Specifically, for the upper arm segment, the system extracts the relative coordinates of the shoulder joint and elbow joint in the initial static standing posture and calculates the Euclidean distance between the two points as the standard length of the upper arm; for the forearm segment, the system calculates the standard length of the forearm using the relative coordinates of the elbow joint and wrist joint; similarly, the lower limbs and other limb segments are measured one by one according to the skeletal topology.

[0046] After obtaining the standard lengths of each limb, the system performs a scaling transformation on the relative coordinates of each key point in each frame. Taking the elbow joint as an example, its relative coordinates represent a spatial vector pointing from the shoulder joint to the elbow joint, and the length of this vector is the actual length of the upper arm in the current frame. The system multiplies each component of this vector by a normalization coefficient (the current actual length divided by the standard length) to obtain the normalized relative position vector of the elbow joint. For example, if the standard length of the target object's upper arm is 0.35 meters, and the actual vector length of the upper arm in the current frame is 0.32 meters, the normalization coefficient is 0.32 / 0.35≈0.914; if the relative coordinates of the elbow joint in the current frame are (0.35, -0.05, 0.02), then the normalized relative position vector is (0.35×0.914, -0.05×0.914, 0.02×0.914)=(0.32, -0.046, 0.018). Through this scaling transformation, the coordinates of all key points are converted into dimensionless relative values ​​based on the patient's own limb length, eliminating the influence of individual body size differences such as height and arm length on motion analysis, and ensuring that subsequent motion assessment is based solely on relative kinematic parameters such as joint angles and trajectory morphology.

[0047] Step S0016: Replace the three-dimensional spatial coordinates in the original keypoint spatiotemporal sequence data with the normalized relative position vectors of each keypoint to generate standardized relative motion data.

[0048] It should be noted that for each keypoint in each frame of data, its timestamp, node ID, and quaternion pose are preserved. The original 3D spatial coordinates (in meters) are replaced with the normalized relative position vector (dimensionless 3D coordinate values) of that keypoint. For example, if the 3D spatial coordinates of the wrist joint in a frame of original data are (0.90, 1.60, 0.10) meters, and after translation and scaling transformations, the normalized relative position vector is (0.875, 0, 0), then after replacement, the coordinates of the wrist joint in that frame become (0.875, 0, 0). After all frames are replaced, standardized relative motion data is generated.

[0049] These operations eliminate interference from individual physiological differences. In the scenario, the system ensures that regardless of the patient's body type, the score is based solely on the standardization of their movements (angle, trajectory shape, etc.), guaranteeing the fairness of the assessment.

[0050] Step S002: Based on standardized relative motion data, calculate the action completion score, action fluency score, and coordination score of each key point when the target object performs the current action, and then weight and fuse them to obtain the current anomalous index of each key point.

[0051] It should be noted that this step is to quantify the quality of the current action and calculate the current anomaly index for each key point.

[0052] The primary indicator of rehabilitation movements is "proper execution." If the standard movement requires 90 degrees of abduction, and the patient can only achieve 60 degrees, it indicates joint stiffness or muscle weakness. This lack of range of motion is direct evidence of inadequate rehabilitation and must be quantified into a specific score to reflect the patient's current upper limit of activity.

[0053] For patients, the actual movements are compared with virtual standard movements in real time: if the maximum range of motion of the standard movement cannot be reached, it indicates that the patient has limited joint mobility or weak muscles and does not yet have the basic ability to complete the movement; if tremors or pauses occur during the movement, it directly disrupts the continuity of the movement, reflecting unstable nerve control or defensive stiffness caused by pain; if a joint fails to reach the standard but other joints are mobilized (such as shrugging the shoulder instead of raising the arm) to forcibly reach the target, this phenomenon can be accurately identified by analyzing the linkage logic between multiple joints. Furthermore, this invention comprehensively evaluates whether the patient's actual movements are standard through three key characteristics: movement completion, movement fluency, and "standardization of multi-joint linkage".

[0054] The calculation of action completion scores at key points when the target object performs the current action, based on standardized relative motion data, further includes steps S21-S23: Step S21: Extract the maximum range of motion of the joints corresponding to each key point when the target object performs the current action from the standardized relative motion data, and record it as the actual peak angle of each key point.

[0055] It should be noted that the system first extracts the sequence of joint motion angles corresponding to each keypoint from the standardized relative motion data. Each keypoint in the standardized relative motion data contains normalized three-dimensional relative coordinates and quaternion poses. Based on the topological relationship of the human skeleton, the system determines three keypoints constituting the angle of each joint: for example, for the shoulder abduction angle, the shoulder, elbow, and hip joints are selected as keypoints, and the shoulder abduction angle is calculated using the angle between the shoulder-to-elbow vector and the shoulder-to-hip vector; for the elbow flexion angle, the shoulder, elbow, and wrist are selected as keypoints, and the angle between the elbow-to-shoulder vector and the elbow-to-wrist vector is calculated. The system traverses all frames of the current action, calculates the motion angle of each joint frame by frame, forms the time sequence of angles for each joint, and takes the maximum value in the sequence as the actual peak angle of the joint corresponding to that keypoint. For example, in a shoulder abduction movement, if the shoulder abduction angle is measured as 82° in frame 15, 88° in frame 23, and 78° in frame 30 during the process of the patient lowering and raising their arm, the system will take the maximum value of 88° as the actual peak angle of the shoulder joint in that movement.

[0056] Step S22: Obtain the ideal peak angle of each key point when the target object performs the current action.

[0057] It should be noted that the ideal peak angle is obtained based on a pre-built standard movement database. This database defines the ideal range of motion for each joint in each rehabilitation movement under standard movement patterns. Data sources include: normal joint range of motion published in rehabilitation medicine guidelines (e.g., shoulder abduction 90°, flexion 120°, elbow flexion 150°, etc.), statistical results of kinematic data collected from healthy individuals performing the same movements (mean or specific percentiles), and adjustable thresholds set by clinical rehabilitation experts based on individual patient conditions. When performing the current movement, the system queries the corresponding ideal peak angle value from the standard movement database based on the movement identifier and key point identifier. For example, for the shoulder joint key point in the "shoulder abduction" movement, the system finds an ideal peak angle of 90°; for the elbow joint key point in the "elbow flexion" movement, the system finds an ideal peak angle of 150°. Meanwhile, the system allows rehabilitation therapists to fine-tune the ideal peak angle before training based on the patient's specific condition, postoperative stage, and individual differences. For example, for patients with severe joint stiffness, a lower target value can be set temporarily (such as shoulder abduction initially at 60°), and then gradually increased to the standard value after function improves, thereby achieving progressive and personalized rehabilitation training goals.

[0058] Step S23: For each key point, calculate the absolute value of the difference between its actual peak angle and the ideal peak angle, divide the absolute value of the difference by the ideal peak angle to obtain the relative deviation ratio, and subtract the relative deviation ratio from 1 to obtain the action completion score of the key point.

[0059] It should be noted that if the action completion score is less than 0, it will be truncated to 0; if it is greater than 1, it will be truncated to 1.

[0060] The scores for the completion of all key points are combined into a sequence of completion scores, with the sequence length equal to the number of key points. Each score ranges from 0 to 1. Low scores directly reflect limited joint mobility, suggesting the need for enhanced stretching or strength training.

[0061] Among them, based on standardized relative motion data, the motion smoothness score of each key point when the target object performs the current action is calculated, which further includes steps S021-S025: Step S021: Extract the instantaneous velocity of each key point in each frame when the target object performs the current action from the standardized relative motion data.

[0062] It should be noted that postoperative patients often experience tremors, pauses, or sudden accelerations in their movements due to weakened muscle control or pain avoidance. This lack of fluidity not only reduces training efficiency but may also indicate that neuromuscular coordination has not been restored. Attempting high-difficulty movements with this tremor greatly increases the risk of secondary muscle strain.

[0063] The system first reads the three-dimensional relative coordinate sequence of each key point in each frame from the standardized relative motion data. The coordinates in the standardized relative motion data have been converted into dimensionless relative values ​​based on the patient's own limb length through translation and scaling transformation, eliminating the influence of individual body size differences. The system uses the central difference method to calculate the instantaneous velocity of each key point: for intermediate frames, the coordinate difference of the key point in the next frame and the previous frame is divided by twice the inter-frame time difference; for the first and last frames, forward difference or backward difference is used for calculation. Specifically, assuming a sampling frequency of 90Hz and an inter-frame time difference Δt≈0.0111 seconds, for the coordinates Pi=(xi, yi, zi) of the key point in the i-th frame, its instantaneous velocity Vi=(P{i+1}-P{i-1}) / (2Δt). For example, the normalized coordinates of the wrist joint in frame 5 are (0.875, 0.00, 0.00), in frame 6 they are (0.890, 0.02, 0.01), and in frame 4 they are (0.860, -0.01, 0.00). Therefore, the instantaneous velocity in frame 5 is ((0.890-0.860) / 0.0222, (0.02-(-0.01)) / 0.0222, (0.01-0.00) / 0.0222)≈(1.35, 1.35, 0.45) (dimensionless units / second). For joint angles, the system extracts angular velocity from the angle time sequence. For example, it calculates the difference in angular velocity between adjacent frames from the shoulder joint abduction angle sequence and divides it by the time interval to obtain the instantaneous angular velocity of that joint (unit: degrees / second). The instantaneous velocity values ​​of all key points constitute the velocity vector sequence of each frame, which serves as the basic input for subsequent acceleration and jerk calculations.

[0064] Step S022: For two adjacent frames, subtract the instantaneous velocity of the corresponding key point in the previous frame from the instantaneous velocity of each key point in the next frame, and then divide by the time difference between frames to obtain the acceleration of each key point in the next frame.

[0065] It should be noted that the acceleration calculation starts from the second frame (because the first frame has no velocity from the previous frame). For the first frame, the acceleration can be set to zero or ignored.

[0066] Step S023: Obtain the acceleration of each key point in the previous frame, subtract the acceleration in the previous frame from the acceleration in the next frame, and then divide by the time difference between frames to obtain the jerk of each key point in the next frame, forming a jerk sequence.

[0067] It should be noted that the jerk calculation starts from the third frame (because the first two frames cannot form an acceleration difference). For the first two frames, the jerk can be set to zero or ignored.

[0068] Step S024: Perform a sliding window average on the jerk sequence, and use the average jerk within the window as the smoothed jerk of each key point in that frame.

[0069] It should be noted that the system performs sliding window averaging on the jerk sequence generated in step S023 to suppress the interference of high-frequency noise on the smoothness assessment and extract the core features reflecting the smoothness of the motion. The size of the sliding window is preset according to the sensor sampling rate and motion type, usually taking the number of frames corresponding to 0.2 to 0.5 seconds. For example, when the sampling rate is 90Hz, the window size can be set to 18 to 45 frames. For each frame, the system takes several frames before and after it (or takes a forward window) to form a window set, and calculates the average of the absolute values ​​of the jerk of all frames in the window (if the jerk is a vector, its magnitude is calculated first), which is used as the smoothed jerk of that frame. For the boundary frames at the beginning and end of the sequence, if the number of effective frames in the window is insufficient, symmetrical padding or averaging using only effective frames can be used to ensure that the length of the output sequence is consistent with the number of original frames.

[0070] Step S025: Calculate the value of an exponential function with the natural constant e as the base and the product of the negative attenuation coefficient and the smoothed jerk as the exponent to obtain the instantaneous smoothness score of the key point in each frame; take the average of the instantaneous smoothness scores of all frames in the entire motion process as the motion smoothness score of the key point.

[0071] It should be noted that the system substitutes the smoothed jerk into the exponential decay function to calculate the motion smoothness score. The specific formula is as follows:

[0072] Where S is the instantaneous smoothness score of the keypoint in each frame, e is the natural constant, a is the preset attenuation coefficient, and j is the smoothed jerk. The attenuation coefficient 'a' is pre-calibrated through experiments or expert experience, and its function is to control the sensitivity of the jerk to the smoothness score: the larger the value of 'a', the more severe the penalty for motion jitter. Usually, a value is selected by comparing and analyzing the motion data of healthy people and postoperative patients, such as a=0.1, where the score of healthy people is close to 1 and the score of patients decreases significantly. This coefficient can be preset by the system or dynamically adjusted according to the patient's recovery stage under the monitoring of the rehabilitation therapist. If the smoothed jerk is a three-dimensional spatial vector (such as calculated from the keypoint position), its magnitude (Euclidean norm) is calculated first and then substituted into the formula; if it is an angular jerk scalar (such as calculated from the joint angle), the scalar value is used directly.

[0073] The system calculates the instantaneous smoothness score for each keypoint frame by frame, and then takes the average score of all frames during the entire action as the smoothness score for that keypoint. If the score of a frame is greater than 1, it is truncated to 1; if it is less than 0, it is truncated to 0, ensuring that the score range is between 0 and 1.

[0074] This exponential function ensures that when the jerk is close to 0 (smooth motion), the smoothness score approaches 1; when the jerk increases significantly (motion jitter, stuttering), the smoothness score rapidly decays to close to 0, thus achieving a quantitative assessment of motion smoothness.

[0075] A low score indicates muscle tremors, suggesting current muscle fatigue or insufficient control, which is a potential risk signal for exercise.

[0076] Among them, based on standardized relative motion data, the coordination score of each key point when the target object performs the current action is calculated, which further includes steps S0021-S0026: Step S0021: Extract the active key point displacement trajectory data and the compensatory key point displacement trajectory data of the target object when performing the current action from the standardized relative motion data.

[0077] It should be noted that patients often "cheat" to achieve a certain range of motion, such as excessively shrugging their shoulders (elevating the cervical spine) or twisting their waist (rotating the pelvis) when raising their arms. While this compensatory behavior may temporarily achieve the desired range of motion, it can lead to strain on non-target muscle groups and even trigger new pain points (such as cervicobrachial syndrome). The system must identify this erroneous pattern of coordination and determine it as an ineffective or even harmful movement.

[0078] The system first retrieves the list of active key points and the list of compensatory key points corresponding to the current training movement from a pre-defined movement definition library. The movement definition library is pre-configured by rehabilitation experts. For each standard rehabilitation movement (such as shoulder abduction or combing hair), it identifies the joints that should primarily move to complete the movement as active key points (e.g., the shoulder joint), and also marks abnormal compensatory joints that the patient may use to mask insufficient range of motion as compensatory key points (e.g., cervical spine, lumbar spine / pelvis). From the standardized relative motion data, the system extracts the three-dimensional coordinate temporal sequence of each key point within the start and end time periods of the movement, according to the node ID of these key points. The active key point displacement trajectory data records the spatial position changes of the active joints in each frame, used for subsequent assessment of their movement regularity; the compensatory key point displacement trajectory data records the spatial position changes of the compensatory joints in each frame, used to detect whether the patient exhibits abnormal compensatory behavior. For example, for the "shoulder abduction" movement, the system extracts the displacement trajectory of the shoulder joint as the active key point, and at the same time extracts the displacement trajectory of the cervical spine and pelvis as the compensatory key points, forming a multi-dimensional time series data array, which provides the basic input for subsequent cumulative displacement calculation and synergy score evaluation.

[0079] Step S0022: For each active key point, determine at least one corresponding compensatory key point.

[0080] It should be noted that the system establishes a mapping relationship between active key points and compensatory key points through a pre-defined motion definition library. For each standard rehabilitation motion, the system explicitly lists the joint combinations that may produce abnormal compensation when performing the motion. Specifically, for each active key point (such as the shoulder joint), the system queries the motion definition library for its corresponding list of compensatory key points (such as the cervical spine, lumbar spine / pelvis). Through this pre-defined mapping mechanism, the system can accurately locate the set of compensatory key points that need to be monitored for each active key point.

[0081] Step S0023: Calculate the cumulative displacement of each compensation key point of the target object from the start of the current action to the end of the current action.

[0082] It should be noted that the system first identifies the start and end frames of the current action. The start frame is typically defined as the moment when the active keypoint first experiences sustained displacement after the action start command is issued; the end frame is defined as the moment when the action is completed or the system determines that the action has terminated. For each compensatory keypoint, the system obtains its three-dimensional coordinate sequence across all frames from the start to the end frame. The cumulative displacement is calculated using a path accumulation method: starting from the start frame, the Euclidean distance between the current frame and the previous frame is calculated frame by frame, and all inter-frame distances are summed to obtain the total movement path length of the compensatory keypoint during the action. For example, for the cervical spine as a compensatory keypoint, if during shoulder abduction, its vertical coordinate (y-value) gradually rises from 0.00 in the start frame to 0.03 in the 10th frame, and then falls back to 0.01 in the end frame, the cumulative displacement should be calculated by adding the absolute value of the vertical change (or three-dimensional spatial distance) between each frame to reflect its actual total movement.

[0083] Step S0024: Divide the cumulative displacement by the preset compensation tolerance threshold to obtain the violation degree ratio of the compensation key point.

[0084] It should be noted that the preset compensation tolerance thresholds are pre-configured in the system database based on biomechanical research and clinical rehabilitation experience. For each standard rehabilitation movement, the system defines the maximum allowable range of motion for each compensation key point, addressing potential abnormal compensation patterns. For example, for the common "shoulder shrugging" compensation in shoulder abduction movements, the preset tolerance threshold for cumulative vertical displacement of the cervical spine is 5 cm; for "trunk tilting" compensation, the preset tolerance threshold for cumulative horizontal rotation of the pelvis is 10 degrees. The thresholds are derived from joint displacement data collected from healthy individuals performing the same movements, using the 95th percentile as the upper limit, and from personalized adjustments by rehabilitation experts based on the patient's specific condition, postoperative stage, and individual differences. All thresholds are stored in the database using movement identifiers, active key points, and compensation key points as primary keys. The system automatically reads the corresponding threshold when executing the current movement and supports real-time dynamic adjustments by rehabilitation therapists during training without modifying the underlying code, thus balancing the scientific rigor of the assessment standards with the flexibility of clinical application.

[0085] If the violation rate exceeds 1, then it is set to 1.

[0086] Step S0025: Subtract the proportion of violation from 1 to obtain the synergy score of the compensation key point.

[0087] Step S0026: If the current active key point involves multiple compensatory key points, then take the minimum value of the synergy score of all compensatory key points as the synergy score of the active key point.

[0088] It should be noted that if the current active key point corresponds to only one compensatory key point, then the synergy score of the compensatory key point is directly used as the synergy score of the active key point.

[0089] Combine the synergy scores of all active key points into a key point synergy score sequence, ranging from 0 to 1.

[0090] A low score indicates an incorrect power application pattern. If not corrected, this pattern will become entrenched and extremely difficult to change, and it can easily lead to injury.

[0091] The weighted fusion yields the current anomalous index for each key point, specifically including: The weighted sum of the action completion score, action fluency score, and coordination score of each key point is calculated to obtain the positive comprehensive score of each key point; 1 is subtracted from the positive comprehensive score to obtain the current anomaly index of each key point.

[0092] It's important to note that a single-dimensional score is insufficient to summarize the overall condition of a joint. A joint might have sufficient range of motion but experience significant shaking, or it might be very stable but suffer from severe compensation. We need a comprehensive indicator to characterize the joint's current "pathological state" or "degree of unhealthiness." The higher this indicator, the greater the problem the joint has at the current stage, and the higher the risk of progressing to the next stage.

[0093] The system first acquires the scores for movement completion, movement fluency, and coordination at the same key point. Then, it reads a preset weight vector containing three weight values, each corresponding to one of the three scores, with the sum of the three weights being 1. These weight values ​​can be determined through expert questionnaires (such as the Delphi method) or clinical data analysis; for example, the highest weight for movement completion might be set to 0.5, the highest to 0.3 for coordination, and the lowest to 0.2 for fluency. Alternatively, the analytic hierarchy process (AHP) can be used to quantify expert judgment. In the system implementation, the weights are typically preset as configurable parameters, allowing therapists to adjust them according to the patient's current stage of development, such as focusing more on amplitude in the early stages and more on stability in the later stages.

[0094] The system multiplies each score by its corresponding weight and then sums them to obtain the positive composite score for that keypoint. It then performs a reverse transformation to obtain the current anomaly index, which is 1 minus the positive composite score. If the result is less than 0, it is truncated to 0; if it is greater than 1, it is truncated to 1, ensuring the anomaly index ranges between 0 and 1. The above steps are repeated for all keypoints to generate a single-keypoint anomaly index sequence. Each value in this sequence is between 0 and 1; a larger value indicates a higher degree of anomaly at the keypoint, meaning it is more dangerous or deviates more from normal movement patterns.

[0095] Step S003: Based on the standard action of the next stage, calculate the structural centrality weight and mechanical lever weight of each key point, and then weight and fuse them to obtain the importance weight of each key point in the next stage.

[0096] It should be noted that, based on biomechanical topology, we deduce how the current anomaly is transformed into the limitations and misfits of the next stage.

[0097] In complex rehabilitation movements, the joints do not exist in isolation, but rather form a network of connections. The more connections a key point has to other key points in a given movement, the greater its centrality. This means that the point is the pivot of the movement, and even small abnormalities in the area corresponding to that key point can easily lead to significant deviations in the corresponding movement. For example, in the "combing hair" movement, the shoulder joint connects the torso, elbow, and hand; if the shoulder joint fails to make proper adjustments, the entire arm trajectory will deviate. Therefore, it is essential to quantify this structural importance.

[0098] Based on the standard actions of the next stage, the structural centrality weights of each key point are calculated, which further includes steps S031-S035: Step S031: Obtain the key point connection matrix of the target object to perform the next stage standard action; where the rows and columns of the matrix correspond to key points, a matrix element of 1 indicates that there is a direct connection between two key points, and a matrix element of 0 indicates that there is no direct connection between two key points.

[0099] It should be noted that the system first extracts the set of key points and their connections from a pre-defined skeletal topology database based on the type of standard movement in the next stage, and constructs a key point connection matrix. This matrix is ​​a square matrix, predefined based on human anatomy. For different movements, a subset of key points can be extracted to generate a sub-matrix. For example, for upper limb movements, the matrix only contains key points related to the upper limb and trunk, ignoring irrelevant nodes of the lower limb.

[0100] Step S032: Calculate the degree centrality of each key point in the key point connection matrix; where degree centrality is the ratio of the number of key points directly connected to each key point in the matrix to the maximum possible number of connections.

[0101] Specifically, for each keypoint in the matrix, the number of matrix elements with a value of 1 in its row (or column) is counted, which is the number of other keypoints directly connected to that keypoint. This number is then divided by the maximum possible number of connections (total number of keypoints minus 1) to obtain the normalized degree centrality, which ranges from 0 to 1. For example, if a certain action involves three keypoints: shoulder, elbow, and wrist, and the shoulder joint is directly connected to the elbow joint (value 1) but not directly connected to the wrist joint (value 0), then the number of direct connections to the shoulder joint is 1, the maximum possible number of connections is 2, and its degree centrality is 0.5.

[0102] Step S033: Calculate the shortest path for all key point pairs in the key point connection matrix.

[0103] It should be noted that the shortest path algorithm is used to calculate the shortest path between all keypoint pairs. Specifically, the system treats the keypoint connection matrix as an undirected, unweighted graph, where keypoints correspond to graph nodes. A matrix element of 1 indicates that there is an edge between two nodes, and an element of 0 indicates that there is no edge. The system uses a breadth-first search (BFS) algorithm, starting from each keypoint and traversing to calculate the shortest path from that starting point to all other keypoints. The length (number of edges traversed), the number of paths, and the sequence of intermediate keypoints traversed are recorded for each shortest path. Since the graph is unweighted, BFS ensures that the path encountered when first visiting the target node is the shortest path. For example, in a chain structure consisting of four keypoints—cervical spine, left shoulder, left elbow, and left wrist—the shortest path from the cervical spine to the left wrist is cervical spine → left shoulder → left elbow → left wrist, with a path length of 3, and the intermediate nodes are the left shoulder and left elbow. After traversing all keypoint pairs, the system obtains the complete set of shortest paths.

[0104] Step S034: For each key point, calculate the proportion of the number of times the key point is passed in all shortest paths to the total number of shortest paths, and use the proportion as the betweenness centrality of the key point.

[0105] Specifically, for each keypoint, the number of paths passing through that keypoint (as an intermediate node, excluding the start and end points) in all shortest paths is counted. This count is divided by the total number of shortest paths to obtain the ratio, which is the betweenness centrality of that keypoint, ranging from 0 to 1. For example, suppose an upper limb movement involves four keypoints: cervical spine (A), left shoulder (B), left elbow (C), and left wrist (D), connected in the order A—B—C—D, forming a chain-like structure. There are a total of 6 keypoint pairs. The shortest path from A to C is A—B—C, passing through intermediate node B; the shortest path from A to D is A—B—C—D, passing through intermediate nodes B and C; the shortest path from B to D is B—C—D, passing through intermediate node C; the remaining pair of nodes have no intermediate nodes. The total number of shortest paths is 6. Statistically, key point B is visited 2 times, key point C is visited 2 times, and key points A and D are visited 0 times. Therefore, the betweenness centrality is as follows: B is 2 / 6≈0.333, C is 2 / 6≈0.333, and A and D are both 0.

[0106] Step S035: Perform a weighted summation of the degree centrality and betweenness centrality of each key point to obtain the structural centrality weight of each key point.

[0107] It should be noted that the system acquires preset weight coefficients, including a first weight coefficient α (corresponding to degree centrality) and a second weight coefficient β (corresponding to betweenness centrality), satisfying α + β = 1. The values ​​of the weight coefficients reflect the balance between the emphasis on "direct connectivity" and "bridging effect" in structural centrality. Typically, α = 0.6 and β = 0.4 are defaulted, meaning degree centrality accounts for 60% and betweenness centrality accounts for 40%. This ratio can be verified through sensitivity analysis to verify the predictive effect under different weights, or recommended and determined by kinematic experts based on action chain theory. The system multiplies the degree centrality of each keypoint by α and the betweenness centrality by β, then sums the results to obtain the structural centrality weight of that keypoint, with a value ranging from 0 to 1.

[0108] These operations are designed to identify "pivotal joints" in the action network. If a high-weight point goes wrong, it can cause the entire action chain to collapse through the connections, making it a key area for risk control.

[0109] Based on the standard actions of the next stage, the mechanical lever weights of each key point are calculated, which further includes steps S0031-S0033: Step S0031: Obtain the maximum linear displacement or maximum spatial volume swept by the far key points corresponding to each key point during the execution of the next stage standard action of the target object.

[0110] It's important to note that in a recovery movement, the greater the range of motion in the limb connecting a key point to other key points in that movement, the more likely it is to affect that key point. This is because large-amplitude movements imply greater torque and load; if the corresponding part of the key point hasn't recovered well, it can easily lead to further deterioration of that part. For example, a large-amplitude wrist swing driven by an unstable elbow will multiply the pressure on the elbow.

[0111] The system first reads the motion trajectory coordinate sequence of each keypoint from the next-stage standard motion database. This sequence contains the three-dimensional spatial coordinates (unit: meters) of the keypoint in each frame during motion execution. For each keypoint, the system determines its corresponding distal keypoint based on the human skeletal topology and the kinetic chain characteristics of the current motion: for example, for the elbow joint, its distal keypoint is usually the wrist joint or fingertip; for the shoulder joint, its distal keypoint may include the elbow joint, wrist joint, or fingertip, depending on the type of motion (e.g., the fingertip is the distal end of the shoulder joint in a combing motion, and the elbow joint in an abduction motion). The system then calculates the maximum linear displacement or maximum spatial volume swept by the distal keypoint during the motion: the maximum linear displacement is calculated using Euclidean distance to determine the maximum distance between any two frames in the trajectory of the distal keypoint; the maximum spatial volume is calculated by performing convex hull calculation on the trajectory point cloud of the distal keypoint to obtain the three-dimensional spatial volume enclosed by the trajectory. The system selects one method based on the required calculation accuracy and the characteristics of the motion; for example, the maximum linear displacement is preferred for actions with strong linearity, while the maximum spatial volume is preferred for multi-directional complex motions.

[0112] Step S0032: Divide the maximum linear displacement or maximum spatial volume swept by the remote key point corresponding to each key point by the maximum value of that value among all key points to obtain the lever ratio of each remote key point.

[0113] Specifically, the leverage ratio of each distal keypoint is obtained by dividing the maximum linear displacement (or maximum spatial volume) of each distal keypoint by the maximum value of that value among all distal keypoints, with a value ranging from 0 to 1. For example, in the combing motion, the distal keypoints corresponding to the shoulder joint are the fingertips, wrist joint, and elbow joint. Assuming the maximum linear displacement of the fingertips is 0.85 meters, the wrist joint is 0.72 meters, and the elbow joint is 0.68 meters, then the leverage ratio of the fingertips is 0.85 / 0.85 = 1.00, the wrist joint is 0.72 / 0.85 ≈ 0.85, and the elbow joint is 0.68 / 0.85 ≈ 0.80. This result indicates that in the standard combing motion, the shoulder joint (driving the fingertips) bears the greatest load, followed by the elbow joint, with the wrist joint bearing a relatively smaller load.

[0114] Step S0033: If a key point corresponds to multiple remote key points, then take the maximum value among all the leverage ratios of the remote key points as the mechanical leverage weight of that key point.

[0115] If a key point corresponds to multiple distal key points (such as the shoulder joint corresponding to the elbow, wrist, and fingertips), then the leverage ratio of each distal key point is calculated separately, and the maximum value is taken as the mechanical leverage weight of that key point (shoulder joint); if there is only one distal key point, then the leverage ratio of that distal key point is taken as the mechanical leverage weight of that key point.

[0116] Generate mechanical lever weight data for all key points, ranging from 0 to 1.

[0117] These operations identified "high-load joints" in the motion. Points with high weights require driving greater mass or amplitude, and if their condition is poor, they are prone to deterioration due to overload.

[0118] The risk of a joint in the next phase depends both on its pivotal position (centrality) within the network and on the load it needs to bear (leverage effect). Only by combining these two factors can the "vulnerability" and "criticality" of that joint in the next phase of action be accurately defined.

[0119] The system obtains preset importance allocation coefficients, including a first allocation coefficient (corresponding to structural centrality) and a second allocation coefficient (corresponding to mechanical leverage weight), the sum of which is 1. These allocation coefficients can be set according to rehabilitation theory or movement type: for example, in the early rehabilitation stage, structural centrality (pivot joints) is more critical and can be assigned a higher weight; in the later strengthening stage, mechanical load is more important and the mechanical leverage weight can be increased; generally, both can be set to 0.5 by default, but leverage weight can be increased for upper limb movements, or centrality weight can be increased for trunk movements. For each key point, the system multiplies its structural centrality weight by the aforementioned first allocation coefficient and its mechanical leverage weight by the aforementioned second allocation coefficient, then sums the results to obtain the next stage importance weight for that key point, with a value ranging from 0 to 1. This importance weight serves as a bridge connecting the current state and future risks; the higher the weight, the more crucial the key point is in the next stage of movement, and the more easily even minor abnormalities can significantly impact the rehabilitation effect.

[0120] Step S004: Using the current anomaly index and the importance weight of each key point in the next stage, obtain the limitation index of each key point in the next stage.

[0121] It should be noted that the greater the anomaly of a key point in the previous stage, the more likely it is to cause significant deviations in the recovery actions related to that key point in the next stage. Furthermore, the greater the anomaly of a key point in the previous stage, the more likely it is to influence the relevant actions in the next stage, leading to further deterioration of the corresponding area. Therefore, we multiply the current "anomaly" by the future "importance weight" to obtain the true "limitations" of that point in the next stage.

[0122] Specifically, it includes: Multiply the current anomalous index of each key point by the importance weight of the key point in the next stage to obtain the limitation index of each key point in the next stage under a single action.

[0123] If the next stage contains multiple actions, the current anomalousness index of each key point in each action is multiplied by the importance weight of that key point in the next stage under that action, and then summed and divided by the number of actions to obtain the limitation index of each key point in the next stage.

[0124] It should be noted that the limitation index will remain between 0 and 1 in the next stage.

[0125] High limitations mean that the joint is very likely to become a bottleneck or injury point in the next stage, which is the core basis for decision-making.

[0126] Step S005: Based on the next-stage limitation index of each key point, calculate the prediction completion rate of each action in the next stage, and then calculate the total stage adaptability score.

[0127] It's important to note that the success of an action often depends on its weakest link (the "weakest link" principle). If a key point on which an action relies has a significant limitation, then no matter how well other joints perform, the overall action cannot be completed to a high standard, or even at all. We need to deduce the expected performance of a specific action from the limitations of the joints.

[0128] Step S005 further includes steps S0051-S0055: Step S0051: For each action in the next stage, obtain the maximum value of the next stage limitation index of all key points involved in the action, and subtract the maximum value from 1 to obtain the predicted completion degree of the action.

[0129] It should be noted that the result ranges from 0 to 1, with higher values ​​indicating better action completion quality. If the action does not depend on any key points, the predicted completion rate is directly set to 1.

[0130] Repeat the above steps for each action in the next stage to obtain a set of action prediction completion scores.

[0131] In the context of the scenario, this set indicates the extent to which the patient is likely to be able to perform this specific action if they were to advance to a higher level now (e.g., only be able to complete 30% of the standard action).

[0132] The next stage of training typically involves multiple movements. A patient's ability to advance should not be judged solely by their performance on a single movement, but rather by the number of movements that are safely achievable within the overall next stage plan. If the predicted completion rate for most movements is low, it indicates that the patient is not yet ready overall, and forcing them to advance carries extremely high risks.

[0133] Step S0052: Count the number of actions with a predicted completion rate greater than or equal to the preset completion rate threshold among all actions in the next stage, and use this number as the number of qualified actions.

[0134] It should be noted that the preset completion threshold is used to determine whether a single movement meets the effective training standard. This threshold is usually set to 0.6, meaning that a predicted completion rate of 60% or higher is considered a qualified movement. This setting is based on clinical rehabilitation practice; when the completion rate is less than 60%, it is generally considered that the patient has not effectively mastered the key points of the movement, and the training effect is poor. In practical applications, this threshold can be optimized and adjusted by analyzing the correlation between a large amount of patient training data and the final rehabilitation effect. For example, the optimal cutoff value can be determined using receiver operating characteristic (ROC) curves to balance training safety and rehabilitation efficiency.

[0135] Step S0053: Divide the number of qualified actions by the total number of actions in the next stage to obtain the pass rate.

[0136] Step S0054: Multiply the predicted completion rate of each action in the next stage by the difficulty weight of the corresponding action, and calculate the weighted average to obtain the expected performance score.

[0137] It should be noted that each movement's difficulty weight corresponds to a movement in the next stage of the training plan, reflecting the movement's importance in the overall training. The determination of movement difficulty weights is based on several criteria: rehabilitation experts assign higher weights to key movements based on clinical experience (e.g., core muscle training movements are weighted at 0.3, and auxiliary movements at 0.1); movements are graded according to their technical difficulty, with higher difficulty resulting in higher weights; or, based on rehabilitation stage theory, key movements for specific functional recovery are assigned higher weights at different stages. The sum of all movement difficulty weights is 1 to ensure the weighted average is between 0 and 1.

[0138] When calculating the expected performance score, a weighted average formula is used: multiply the predicted completion rate of each action by its corresponding difficulty weight and then sum the results to obtain the expected performance score. Since the sum of the difficulty weights of each action is 1, this weighted sum is the weighted average.

[0139] Step S0055: Obtain the preset first fusion coefficient and second fusion coefficient, multiply the pass rate by the first fusion coefficient, multiply the expected performance score by the second fusion coefficient, sum them, and then multiply by 100 to obtain the stage adaptability total score.

[0140] Specifically, the pass rate and expected performance score are proportionally combined into a total stage suitability score, with the formula being (first fusion coefficient × pass rate + second fusion coefficient × expected performance score) × 100. The sum of the fusion coefficients is 1, and the final result is rounded down to the nearest integer, ranging from 0 to 100.

[0141] The first and second fusion coefficients represent the proportions of the achievement rate and expected performance score in the total score. They are generally set to 0.5 for both, indicating equal importance. These can be adjusted according to the rehabilitation stage; for example, in the early stages, more emphasis is placed on the achievement rate (safety first), while in later stages, overall performance is given greater importance. The coefficients can be obtained through expert experience or machine learning methods (such as logistic regression).

[0142] Provide the final, advanced feasibility conclusion. A high score indicates safety and efficiency, while a low score indicates danger and ineffectiveness.

[0143] Step S006: Determine whether the target object is allowed to enter the next stage of training based on the overall stage fit score.

[0144] The system's preset judgment thresholds typically include: A safe progression threshold (e.g., ≥70 points): This indicates the patient's ability fully meets the requirements of the next stage and they can safely progress. A borderline threshold (e.g., 50 points): This indicates the patient is in the middle range and requires consideration of other factors (e.g., doctor's assessment, patient's subjective feelings) before making a decision. An unsafe threshold (e.g., <50 points): This indicates the current ability is insufficient to support the next stage; forcibly progressing carries high risk, and training should be postponed or the patient should revert to the current stage for intensive training.

[0145] Threshold acquisition method: By performing receiver operating characteristic curve analysis on the dataset of patients who have completed rehabilitation, the maximum point of Youden's index is used to determine the safety advancement threshold that can best distinguish between "safety advancement" and "excessive risk"; the statistical cut point (such as logistic regression or decision tree model) where the incidence of adverse events increases significantly is used to define the unsafe threshold; and the interval between the two naturally constitutes the critical region that needs to be manually reviewed.

[0146] Output the corresponding result based on the score range: If the score is greater than or equal to the safety threshold for advancement, the result is "advancement recommended" and the reason is recorded: the patient's overall adaptability is high and most actions in the next stage can be completed safely.

[0147] If the score falls between the critical threshold and the safe progression threshold (inclusive), the result is "manual review required," and it is indicated that the patient's completion of some movements is insufficient, and it is recommended that a rehabilitation therapist conduct further evaluation in conjunction with other clinical indicators.

[0148] If the score is below the critical threshold, the result is "postponing further training," and the reason is recorded: the patient's current limitations are significant, and the risk of advancing is high. It is recommended to continue training at the current stage or strengthen weak areas.

[0149] To aid decision-making, the following data may be added: List of actions that did not meet the target (actions whose predicted completion rate is below the completion rate threshold) and their corresponding key point limitation indices.

[0150] The key points with the highest risk, their anomaly index, and their importance weights.

[0151] Recommended training focus (such as intensive practice on key points for low scores).

[0152] The decision result (string), score, and additional information are output together for the therapist or the system to automatically execute the next training plan.

[0153] In summary, this invention not only assesses the immediate rehabilitation effect of the current stage based on multiple dimensions such as movement completion, fluency, and synergy, but also quantifies and predicts the potential impact of current joint abnormalities on the target joint in the next stage through algorithms such as topological centrality and mechanical leverage amplitude. A new stage is unlocked only when the prediction results indicate that the patient has the physiological basis for safe progression; otherwise, the progression is automatically locked and reinforcement training is enforced. This effectively avoids negative interference to recovered joints caused by abnormal compensation induced by higher-order movements, prevents functional regression, and ensures the integrity and safety of the rehabilitation process. Simultaneously, through 135-point whole-body model monitoring and individualized normalization processing, fair and accurate assessment from fine finger movements to large joint compound movements is achieved, significantly improving the adaptability and risk control capabilities of rehabilitation training.

[0154] This invention also proposes a virtual reality rehabilitation training system based on motion contrast feedback; please refer to [link / reference]. Figure 2 The diagram illustrates a block diagram of a virtual reality rehabilitation training system based on motion contrast feedback according to an embodiment of the present invention. The system includes: The data acquisition module 100 is used to acquire the original key point spatiotemporal sequence data and generate standardized relative motion data based on the original key point spatiotemporal sequence data. The original key point spatiotemporal sequence data includes timestamps, node IDs, three-dimensional spatial coordinates and quaternion poses. The node IDs correspond to 135 preset human key points. The data analysis module 200 is used to calculate the action completion score, action fluency score and coordination score of each key point when the target object performs the current action based on standardized relative motion data, and to obtain the current anomalous index of each key point by weighted fusion. Based on the standard actions for the next stage, calculate the structural centrality weight and mechanical lever weight of each key point, and then weight and fuse them to obtain the importance weight of each key point for the next stage. By using the current anomalous index of each key point and the importance weight of the next stage, the limitation index of each key point in the next stage is obtained. Based on the next-stage limitation index of each key point, the predicted completion rate of each action in the next stage is calculated, and then the overall stage adaptability score is calculated. The decision module 300 is used to determine whether the target object is allowed to enter the next stage of training based on the overall stage fit score.

[0155] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the virtual reality rehabilitation training system based on motion contrast feedback and the virtual reality rehabilitation training method based on motion contrast feedback provided in the above embodiments belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0156] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0158] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A virtual reality rehabilitation training method based on motion contrast feedback, characterized in that, The method includes the following steps: The original spatiotemporal sequence data of key points is acquired, and standardized relative motion data is generated based on the original spatiotemporal sequence data of key points. The original spatiotemporal sequence data of key points includes timestamps, node IDs, three-dimensional spatial coordinates and quaternion poses. The node IDs correspond to 135 preset human body key points. Based on standardized relative motion data, the action completion score, action fluency score and coordination score of each key point when the target object performs the current action are calculated, and the weighted fusion is used to obtain the current anomaly index of each key point. Based on the standard actions for the next stage, calculate the structural centrality weight and mechanical lever weight of each key point, and then weight and fuse them to obtain the importance weight of each key point for the next stage. By using the current anomalous index of each key point and the importance weight of the next stage, the limitation index of each key point in the next stage is obtained. Based on the next-stage limitation index of each key point, the predicted completion rate of each action in the next stage is calculated, and then the overall stage adaptability score is calculated. The decision on whether to allow the target subject to proceed to the next stage of training is based on the overall stage fit score.

2. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps involved in acquiring the original spatiotemporal sequence data of key points and generating standardized relative motion data based on the original spatiotemporal sequence data of key points are as follows: Acquire the original RGB-D video frame sequence and the original signal from the IMU inertial sensor of the target object; The original RGB-D video frame sequence and the original signal from the IMU inertial sensor are input into a pre-trained 3D human pose estimation neural network to obtain the original spatiotemporal sequence data of key points. The 3D human pose estimation neural network adopts an encoder-decoder structure. This network extracts the two-dimensional heat map of human key points in the RGB-D video frame through a convolutional neural network, and maps the two-dimensional key points to three-dimensional space by combining the depth map. At the same time, the original signal from the IMU inertial sensor is used to correct and smooth the motion trajectory of the key points through a temporal convolutional network. Finally, the three-dimensional spatial coordinates and quaternion pose of 135 preset human key points in each frame are output. The three-dimensional spatial coordinates of each key point in the original key point spatiotemporal sequence data are transformed into relative coordinates relative to the shoulder joint; Based on the transformed coordinates of each key point, obtain the length of each limb vector; Obtain the standard length of each limb of the target object measured in the initial static posture, divide the length of each limb vector by its corresponding standard length, and obtain the normalized relative position vector of each key point. The three-dimensional spatial coordinates in the original keypoint spatiotemporal sequence data are replaced with the normalized relative position vectors of each keypoint to generate standardized relative motion data.

3. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps involved in calculating the motion completion score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Extract the maximum range of motion of the joints at each key point when the target object performs the current action from the standardized relative motion data, and record it as the actual peak angle of each key point. Obtain the ideal peak angles of key points of the target object when it performs the current action; For each key point, calculate the absolute value of the difference between its actual peak angle and the ideal peak angle. Divide the absolute value of the difference by the ideal peak angle to obtain the relative deviation ratio. Subtract the relative deviation ratio from 1 to obtain the action completion score for that key point.

4. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps involved in calculating the motion fluency score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Obtain the instantaneous velocity of each key point in each frame of the standardized relative motion data when the target object performs the current action; For two adjacent frames, subtract the instantaneous velocity of the corresponding key point in the previous frame from the instantaneous velocity of each key point in the later frame, and then divide by the time difference between frames to obtain the acceleration of each key point in the later frame. The acceleration of each key point in the previous frame is obtained, the acceleration in the next frame is subtracted from the acceleration in the previous frame, and then divided by the time difference between frames to obtain the jerk of each key point in the next frame, forming an acceleration sequence. A sliding window average is applied to the jerk sequence, and the average jerk within the window is used as the smoothed jerk of each key point in that frame. Calculate the value of an exponential function with the natural constant e as the base and the product of a negative attenuation coefficient and smoothed jerk as the exponent to obtain the instantaneous smoothness score of the key point in each frame; take the average of the instantaneous smoothness scores of all frames in the entire motion process as the motion smoothness score of the key point.

5. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps involved in calculating the coordination score of each key point when the target object performs the current action based on standardized relative motion data are as follows: Extract active key point displacement trajectory data and compensatory key point displacement trajectory data of the target object when performing the current action from standardized relative motion data; For each active key point, identify at least one corresponding compensatory key point; Calculate the cumulative displacement of each compensation key point of the target object from the start of the current action to the end of the current action; Divide the cumulative displacement by the preset compensation tolerance threshold to obtain the violation rate of the compensation key point. Subtracting the proportion of violations from 1 yields the synergy score for this key compensation point. If the current active key point involves multiple compensatory key points, then the minimum value of the synergy score of all compensatory key points is taken as the synergy score of the active key point. The weighted fusion process yields the current anomalous index for each key point, and includes the following specific steps: The weighted sum of the action completion score, action fluency score, and coordination score of each key point is calculated to obtain the positive comprehensive score of each key point; 1 is subtracted from the positive comprehensive score to obtain the current anomaly index of each key point.

6. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps for calculating the structural centrality weights of each key point based on the standard actions of the next stage are as follows: Obtain the key point connection matrix for the target object to perform the next stage standard action; where the rows and columns of the matrix correspond to key points, a matrix element of 1 indicates that there is a direct connection between two key points, and a matrix element of 0 indicates that there is no direct connection between two key points; Degree centrality of each keypoint in the keypoint connection matrix; where degree centrality is the ratio of the number of keypoints directly connected to each keypoint in the matrix to the maximum possible number of connections. Find the shortest path for all keypoint pairs in the keypoint connection matrix; For each key point, calculate the proportion of the number of times the key point is visited in all shortest paths to the total number of shortest paths, and use the proportion as the betweenness centrality of the key point. The structural centrality weights of each key point are obtained by weighted summation of the degree centrality and betweenness centrality.

7. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps involved in calculating the mechanical lever weights at each key point based on the standard action of the next stage are as follows: During the execution of the next stage of standard actions by the target object, obtain the maximum linear displacement or maximum spatial volume swept by the remote key points corresponding to each key point; Divide the maximum linear displacement or maximum spatial volume swept by the far key point corresponding to each key point by the maximum value of that value among all key points to obtain the leverage ratio of each far key point. If a key point corresponds to multiple remote key points, then the maximum value among the leverage ratios of all remote key points is taken as the mechanical leverage weight of that key point.

8. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps for obtaining the limitation index of each key point in the next stage by using the current anomaly index and the importance weight of the next stage are as follows: Multiply the current anomalous index of each key point by the importance weight of the key point in the next stage to obtain the limitation index of each key point in the next stage under a single action. If the next stage contains multiple actions, the current anomalousness index of each key point in each action is multiplied by the importance weight of that key point in the next stage under that action, and then summed and divided by the number of actions to obtain the limitation index of each key point in the next stage.

9. The virtual reality rehabilitation training method based on motion contrast feedback according to claim 1, characterized in that, The specific steps for calculating the predicted completion rate of each action in the next stage based on the limitation index of each key point, and then calculating the overall stage adaptability score, are as follows: For each action in the next stage, obtain the maximum value of the next stage limitation index of all key points involved in the action, and subtract the maximum value from 1 to obtain the predicted completion rate of the action. The number of actions with a predicted completion rate greater than or equal to the preset completion rate threshold among all actions in the next stage is counted as the number of qualified actions; Divide the number of qualified actions by the total number of actions in the next stage to obtain the pass rate; The predicted completion rate of each action in the next stage is multiplied by the difficulty weight of the corresponding action, and the weighted average is calculated as the expected performance score. Obtain the preset first fusion coefficient and second fusion coefficient, multiply the pass rate by the first fusion coefficient, multiply the expected performance score by the second fusion coefficient, sum them, and then multiply by 100 to obtain the stage adaptability total score.

10. A virtual reality rehabilitation training system based on motion contrast feedback, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by the processor, it implements the steps of a virtual reality rehabilitation training method based on motion contrast feedback as described in any one of claims 1-9.

Images