Orthopedic rehabilitation action recognition and evaluation method based on image recognition
By combining a parametric biomechanical digital twin model with a graph neural network, rehabilitation postures can be verified and corrected in real time, solving the problems of joint dislocation and false positives in orthopedic rehabilitation and improving the safety and professionalism of rehabilitation training.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DONGGUAN TRADITIONAL CHINESE MEDICINE HOSPITAL
- Filing Date
- 2026-03-24
- Publication Date
- 2026-06-19
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image recognition and intelligent medical technology, specifically relating to a method for recognizing and evaluating orthopedic rehabilitation movements based on image recognition. Background Technology
[0002] With the deep integration of digital healthcare and intelligent rehabilitation technologies, image recognition-based orthopedic movement assessment has become a key technology for assisting patients' postoperative recovery and motor function reconstruction. Modern rehabilitation systems increasingly rely on non-invasive visual sensors to quantitatively evaluate the quality of rehabilitation training through real-time capture and analysis of human movement postures. In high-precision medical scenarios, the system needs to accurately monitor the amplitude, force, and trajectory of patients' limb movements. This requires the recognition algorithm to not only possess strong spatial modeling capabilities but also a deep understanding of the intrinsic connections within the human physiological structure to ensure the professionalism and guidance of the assessment results.
[0003] Topological modeling of skeletal key points using graph neural networks is a current mainstream technological approach, aiming to fit complex rehabilitation movement patterns by extracting spatiotemporal feature sequences. Topological modeling typically treats the human body as a discrete set of geometric coordinates, attempting to establish a mapping relationship between visual representations and standard movement templates, with the goal of providing intuitive data support for rehabilitation physicians. Orthopedic rehabilitation involves the coordinated operation of multiple systems including muscles, bones, and ligaments. Simple image feature fitting often ignores fundamental constraints of human dynamics and biomechanics, resulting in a lack of physiologically sound support for the visually reconstructed movement sequences.
[0004] Existing technologies primarily focus on coordinate fitting of two-dimensional or three-dimensional key points, lacking deep integration with human physical characteristics. This leads to joint dislocations or clipping phenomena that violate physiological principles when handling complex rehabilitation movements. When interference factors such as limb occlusion or clothing covering occur in the rehabilitation scenario, pure visual models often collapse in posture estimation due to loss of spatial information, making it difficult to infer the true state of occluded parts based on mechanical logic. Traditional algorithms only assess the standardization of movements at the morphological similarity level, failing to identify visually similar but physiologically harmful erroneous postures. This results in a risk of false positives in assessment reports, threatening the safety of clinical training. Therefore, an image recognition-based method for orthopedic rehabilitation movement recognition and assessment is needed. Summary of the Invention
[0005] The purpose of this invention is to provide an image recognition-based method for recognizing and evaluating orthopedic rehabilitation movements, which can solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the technical solution adopted by this invention is: an orthopedic rehabilitation movement recognition and assessment method based on image recognition, comprising the following specific steps: Step 1: Acquire a continuous multi-view image sequence of the patient during rehabilitation training using a non-invasive visual sensor array, and extract the three-dimensional spatial coordinate data of key points of the human skeleton from it; Step 2: Construct a parametric biomechanical digital twin model that includes skeletal structure, muscle distribution, and ligament connections. The parametric biomechanical digital twin model has constraints on joint range of motion, muscle force line direction, and ligament tension response mechanism. Step 3: The three-dimensional spatial coordinate data of the key skeletal points are used as driving input and injected into the node structure of the graph neural network, where each node corresponds to a key part of human anatomy and each edge represents the biomechanical connection between adjacent parts. Step 4: During the graph neural network inference process, the biomechanical digital twin model is synchronously invoked to verify the physical feasibility of the current posture. If the recognition result causes the joint angle to exceed the preset physiological activity range or causes ligament overload, the posture correction mechanism is triggered. Step 5: Based on the corrected posture data, calculate the spatiotemporal trajectory similarity between the patient's current movement and the standard rehabilitation movement template, and generate a comprehensive evaluation result by combining the biomechanical rationality score.
[0007] Preferably, the non-invasive visual sensor array in step 1 consists of at least three depth cameras, which are respectively arranged in front of, to the side and above the patient to ensure that complete motion information can still be obtained in the case of limbs occluding each other, and the synchronous acquisition frequency of each camera is not less than 30 frames per second.
[0008] Preferably, in step 2, the parameterized biomechanical digital twin model adopts a layered modeling strategy, with the bottom layer being a rigid skeletal chain structure, the middle layer being a deformable muscle envelope layer, and the top layer being a dynamic ligament tension feedback layer. The three layers interact with each other through preset mechanical coupling rules.
[0009] Preferably, in step 3, the node features of the graph neural network not only include the spatial coordinates of key points, but also integrate local motion velocity vectors and acceleration vectors, and the edge weights are dynamically adjusted according to the biomechanical dependence strength between adjacent joints.
[0010] Preferably, the physical feasibility verification process in step 4 includes joint angular velocity mutation detection, muscle synergistic activation consistency judgment, and ligament strain energy threshold comparison. If any indicator exceeds the preset threshold, it is determined to be a non-physical posture.
[0011] Preferably, the pose correction mechanism in step 4 adopts an inverse dynamics optimization algorithm, which, while keeping the positions of visible key points unchanged, infers the reasonable pose of the occluded part based on the biomechanical digital twin model and updates the internal state of the graph neural network.
[0012] Preferably, the standard rehabilitation movement template in step 5 is stored in the form of a time series, with each moment marked with the corresponding ideal joint angle, center of gravity projection area and muscle force pattern, for multi-dimensional comparison and analysis.
[0013] Preferably, the comprehensive evaluation results include three core indicators: percentage of movement completion, biomechanical safety level, and potential injury risk warning. The biomechanical safety level is divided into multiple predetermined levels based on the degree to which the joint load deviates from the normal physiological range.
[0014] Preferably, the method further includes longitudinal tracking and analysis of multiple rehabilitation training records, identifying the evolution trend of the patient's movement patterns through clustering algorithms, and automatically generating personalized training adjustment suggestions.
[0015] Preferably, the biomechanical digital twin model supports parameter adaptation for patients of different ages, genders, and pathological states. The adaptation parameters include bone mineral density coefficient, muscle elastic modulus, and articular cartilage friction coefficient.
[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. By deeply coupling the biomechanical digital twin model into the inference process of the graph neural network, a motion recognition mechanism with physical common sense constraints is constructed, avoiding joint misalignment and clipping phenomena caused by pure visual fitting; under severe limb occlusion conditions, the system can reasonably infer the posture of invisible parts based on human dynamics logic, improving the robustness and continuity of posture estimation.
[0017] 2. By applying a medically meaningful physiological feasibility check to the recognition results, false positives that are visually similar but actually pose a risk of sports injury are effectively eliminated, enhancing the clinical safety and professional credibility of orthopedic rehabilitation assessment; it realizes a paradigm upgrade from single-morphological matching to multi-dimensional biomechanical compliance evaluation, providing a technical foundation for intelligent rehabilitation systems that combines engineering precision and medical rigor. Attached Figure Description
[0018] Figure 1 The flowchart is based on the present invention; Figure 2 This is a schematic diagram of data flow according to the present invention; Figure 3 This is a flowchart illustrating the process of verifying the physical feasibility of the current posture based on the biomechanical digital twin model according to the present invention, and triggering inverse dynamics optimization correction when a non-physical posture is determined. Figure 4 This is a schematic diagram of the multi-level mechanical coupling and interaction analysis based on the rigid skeletal chain, deformable muscle envelope layer and dynamic ligament tension feedback layer according to the present invention. Figure 5 This is a flowchart illustrating how, according to the present invention, a multi-dimensional comparative analysis based on a standard rehabilitation movement template is performed, combined with longitudinal tracking analysis, to generate personalized training adjustment suggestions. Detailed Implementation
[0019] Example 1: Please refer to the appendix Figure 1 To be continued Figure 5 To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments.
[0020] In the image recognition-based orthopedic rehabilitation movement recognition and assessment method, step 1 involves acquiring a continuous multi-view image sequence of the patient during rehabilitation training using a non-invasive visual sensor array. In specific implementation, the non-invasive visual sensor array is configured to include at least three independent depth camera components. These depth camera components are strategically arranged within the patient's training space, with the first depth camera positioned directly in front of the patient to capture features of the frontal posture and core interference region. The second depth camera is located to the left or right of the patient, with its optical axis forming an angle between 45 and 90 degrees with the optical axis of the front camera, and is used to acquire lateral depth information; the third depth camera is mounted on a hanging bracket above the patient, looking down over the training area to provide spatial reference in the vertical dimension.
[0021] This spatial layout ensures that even when some limbs obstruct each other during complex rehabilitation movements (such as knee flexion, chest expansion, or compound joint rotation), the system can still obtain complete motion information through a multi-view data completion mechanism.
[0022] During the data acquisition phase, the acquisition frequency of each depth camera component is strictly set to no less than 30 frames per second to ensure the continuity of motion capture. The cameras are synchronized at the nanosecond level via hardware synchronization trigger signals or network time protocols. The acquired raw image data stream first passes through a preprocessing module, which performs operations including background culling, illumination compensation, and depth map denoising.
[0023] Using a pre-trained deep learning-based human skeleton detection model, key anatomical nodes of the human body are identified from multi-view images in each frame. Through multi-view geometric projection, the coordinates of two-dimensional key points from different perspectives are mapped to a unified global three-dimensional coordinate system, thereby extracting the three-dimensional spatial coordinate data of the key points of the human skeleton. This data constitutes a time-varying three-dimensional coordinate sequence, completely recording the physical trajectory of rehabilitation training.
[0024] In step 2, this invention constructs a parametric biomechanical digital twin model that includes skeletal structure, muscle distribution, and ligament connections. The construction of this parametric biomechanical digital twin model is based on a hierarchical modeling strategy, ensuring in-depth simulation from microscopic mechanical response to macroscopic motion performance. The bottom layer is a rigid skeletal chain structure. In this bottom layer, the human skeleton is abstracted as a series of rigid links with fixed lengths and mass distribution characteristics, and these links are connected by kinematic pairs with specific degrees of freedom.
[0025] For example, the shoulder joint is modeled as a ball-and-socket joint with three degrees of freedom, while the elbow joint is modeled as a rotational joint with one major degree of freedom. The range of motion of each joint is assigned strict physiological constraints, which form the basis for subsequent physical feasibility verification. The middle layer is a deformable muscle envelope. This middle layer simulates human muscles by covering the skeletal chain with a geometrically structured mesh possessing biophysical properties. Each muscle group (such as the biceps brachii and quadriceps femoris) is assigned specific force direction constraints. These constraints define the point of application and direction of the force vector generated by the muscle during contraction or stretching.
[0026] The muscle model also incorporates cross-sectional area parameters and contractile stress coefficients to simulate muscle morphological changes and the resulting biomechanical support forces during specific movements. The upper layer is a dynamic ligament tension feedback layer. Ligaments are abstracted as elastic units connecting adjacent bones, possessing a non-linear tension response mechanism. When joint displacement causes changes in the linkage spacing, the system calculates the simulated ligament strain. If the strain exceeds a preset elastic limit, the model generates a tension feedback signal.
[0027] These three layers interact with each other through pre-defined mechanical coupling rules: changes in the skeletal chain pose drive deformation of the muscle envelope layer, while simultaneously pulling on the ligament tension layer to generate stress; conversely, the tension of the ligaments and the mechanical constraints of the muscles also feed back to the skeletal chain, limiting its movement beyond normal anatomical limits. The digital twin model supports parameter adaptation. The system automatically adjusts model parameters for patients of different ages, genders, and pathological conditions.
[0028] For elderly patients, the system lowers the bone mineral density and muscle elastic modulus, and increases the friction coefficient of articular cartilage. These adaptation parameters are derived from the patient's clinical imaging reports, body composition analysis data, and medical history records.
[0029] In step 3, the three-dimensional spatial coordinate data of the skeletal key points are used as driving inputs and injected into the node structure of the graph neural network. The topology of the graph neural network is designed to strictly correspond to the human anatomical structure, where each node represents a specific key anatomical location (such as the center of the left knee, the right ankle joint, etc.), and each edge between nodes represents the biomechanical connection relationship between adjacent locations. This relationship mathematically simulates the linkage logic of human joints.
[0030] During the feature extraction stage, the feature vector of each node not only contains the three-dimensional spatial coordinates at the current moment, but also integrates the local motion velocity vector and acceleration vector through differential operations on continuous frame data. This fusion of multi-dimensional features enables the network to perceive the dynamic characteristics of actions, such as explosive force, pauses, or tremors. The weights of the edges are not fixed values, but are dynamically adjusted according to the biomechanical dependence between adjacent joints.
[0031] For example, during a squat, the edge weights between the knee and ankle joints increase, reflecting their strong coupling in mechanical transmission. In each layer of a neural network, message passing between nodes follows biomechanical pathways, ensuring that the movement state of the distal limbs is effectively fed back to the proximal core via the kinetic chain.
[0032] In step 4, during the graph neural network inference process, the system simultaneously invokes the biomechanical digital twin model to verify the physical feasibility of the current posture. This verification process is a real-time monitoring system with multiple indicators.
[0033] First, there's the joint angular velocity mutation detection: the system calculates the rate of change of joint angles between adjacent frames in real time. If the physical acceleration corresponding to the rate of change of joint angles between adjacent frames is not achievable in human physiological structure (such as an instantaneous jump in angular displacement), it is determined to be an anomaly caused by sensor noise or algorithm fitting error. Second, there's the muscle co-activation consistency judgment: based on the muscle force line direction constraints in the digital twin model, it analyzes whether the muscle groups involved in the movement under the current posture conform to the co-force logic.
[0034] For example, if the model detects that the biceps and triceps simultaneously generate abnormally strong antagonistic contractile forces when the arm is flexed, it determines that there is a logical problem with the posture. Finally, there is a comparison of ligament strain energy thresholds: the system calculates the simulated strain energy of ligaments in various parts of the body under the current posture. If the energy value exceeds the preset safety threshold, it means that the current posture may lead to ligament tearing or dislocation, and is determined to be a non-physical posture.
[0035] Once a non-physical posture is determined, the system will immediately trigger the posture correction mechanism. The posture correction mechanism employs an inverse dynamics optimization algorithm. During the correction process, the system first anchors the locations of visible key points with extremely high confidence, acquired by the sensor array. For occluded or misidentified parts, the algorithm uses a biomechanical digital twin model as a hard constraint to find the optimal posture solution within the feasible solution space.
[0036] This optimal solution must satisfy the condition that all joint angles are within a preset physiological range, and the biomechanical potential energy of the whole body is in a reasonable local minimum state. In this way, the system can reasonably infer the pose of the occluded part based on human dynamics logic, and update the internal hidden state of the graph neural network using the corrected coordinate values, thus eliminating the collapse at the visual recognition level.
[0037] In step 5, based on the corrected posture data, the system performs the final recognition and evaluation task. First, the system calculates the spatiotemporal trajectory similarity between the patient's current movement and the standard rehabilitation movement template. The standard rehabilitation movement template is stored in the system database in time-series format. The rehabilitation movement template not only contains the three-dimensional sequence of ideal joint angles at each moment, but also details the movement trajectory of the center of gravity projection area on the support surface, as well as the muscle force exertion pattern parameters under ideal conditions.
[0038] The similarity calculation employs the logic of dynamic time warping: by finding the matching path that minimizes the cumulative distance between the patient's movement sequence and the standard template sequence, the spatial difference between the two after time axis alignment is calculated. The system then combines this with the biomechanical rationality score generated in step 4. This score reflects the degree of deviation between the patient's actual joint load, predicted muscle fatigue, and energy expenditure rate from medical standards during the movement.
[0039] The system generates a comprehensive assessment result. This result includes three core indicators: First, the percentage of movement completion, calculated by determining the proportion of the patient's actual trajectory that covers the standard trajectory; second, the biomechanical safety level, which is categorized into several predetermined levels (e.g., safe, low-risk, medium-risk, high-risk) based on the degree to which joint load deviates from the normal physiological range, with each level corresponding to a different load threshold range; and third, potential injury risk warnings, where the system accurately identifies which joints have been subjected to inappropriate shear forces or excessive pressure during movement and provides specific anatomical location suggestions.
[0040] Furthermore, the system includes longitudinal tracking and analysis of multiple rehabilitation training records. By processing the feature vectors of each patient's training sessions using clustering algorithms, the system can identify the evolutionary trends of the patient's movement patterns. For example, the system can detect a steady increase in the maximum knee flexion angle or an improvement in gait symmetry during two consecutive weeks of training. Based on these trends, the system automatically generates personalized training adjustment suggestions, such as increasing the intensity of strength training for specific muscle groups or adjusting the number of repetitions of movements, thereby achieving dynamic optimization of the rehabilitation program.
[0041] Example 2: In another specific implementation scenario, the image recognition-based orthopedic rehabilitation movement recognition and assessment method described in this invention is applied to the monitoring of scoliosis correction training. In this example, the biomechanical digital twin model in step 2 undergoes specific parameter enhancement.
[0042] To address the physiological characteristics of the spine, the skeletal chain structure in the parametric biomechanical digital twin model was refined into a complex multi-link model comprising 24 vertebrae and the sacrum. Each vertebra is connected by a flexible joint with six degrees of freedom to simulate the compression, stretching, lateral flexion, and rotation characteristics of the intervertebral disc. Within the muscle envelope, the mechanical modeling of the erector spinae, psoas major, and external oblique muscles was particularly enhanced, with their contractile force parameters individually calibrated based on the patient's postoperative MRI data.
[0043] In the graph neural network construction in step 3, nodes are not only distributed at the major limb joints, but also a high density of observation points is added at the spinous processes of the spine. The edge weight setting logic is adjusted to focus on trunk balance. When the patient performs lateral bending rehabilitation movements, the graph neural network will focus on extracting the relative angular changes between different segments of the spine.
[0044] In the physical feasibility verification step 4, the system introduces a center of gravity balance constraint. By calculating the position of the human body's center of mass in the current posture and projecting it onto the support surface, if the projected center of gravity exceeds the edge of the support area formed by the two feet, and the model detects that the muscles cannot provide sufficient anti-overturning moment, the system will determine that the human posture is an unsteady artifact and trigger posture correction based on equilibrium mechanics. This correction not only corrects the position of key points but also redistributes the distribution of the body's center of gravity to conform to basic physical equilibrium laws.
[0045] In the assessment phase of step 5, the standard movement template was expanded to include comparisons of spinal curvature parameters. The comprehensive assessment results further refined the safety level into a prediction of intervertebral pressure. If the intervertebral pressure exceeds the patient's pathological bearing limit at any given moment, the system will issue an immediate vibration alert via the terminal device. Longitudinal tracking analysis pays particular attention to the changing trends of scoliosis (Cobb angle) during dynamic movement, providing physical therapists with precise data support.
[0046] Example 3: In implementation scenarios involving telemedicine and home rehabilitation, the method of the present invention demonstrates adaptability to low-bandwidth environments and heterogeneous computing devices.
[0047] In step 1, the non-invasive visual sensor array can consist of consumer-grade depth cameras suitable for home use. To compensate for the accuracy limitations of low-cost sensors, a time-series-based smoothing filtering algorithm is added during image preprocessing to filter out high-frequency jitter. In step 2, considering the computational capabilities of the home rehabilitation patient device, the biomechanical digital twin model employs a multi-level accuracy switching mechanism. When sufficient computational resources are detected, a high-precision model containing the entire muscle envelope is activated; when computational resources are limited, the model automatically downgrades to a simplified version containing only skeletal chain and core ligament constraints, while retaining key joint range of motion limitations to ensure a safety baseline.
[0048] In step 3, the inference task of the graph neural network is distributed between edge devices and cloud servers. The edge devices are responsible for extracting basic 3D coordinates and performing preliminary physical feasibility checks (such as joint hyperextension detection), while complex biomechanical coupling calculations and inverse dynamics optimization are completed in the cloud. This distributed architecture ensures the real-time nature of the evaluation; even with fluctuating network latency, the local physical constraint layer can provide real-time safety alerts.
[0049] In step 4, to address the decreased recognition rate caused by complex backgrounds and uneven lighting in home environments, the pose correction mechanism enhances the prediction function based on historical motion patterns. The system utilizes long short-term memory logic combined with a biomechanical digital twin model to perform probabilistic pose prediction for occluded limbs. For example, during arm-swinging exercises, if the arm completely enters the torso occlusion area, the system predicts its trajectory within the occlusion area based on the arm's velocity vector, acceleration vector, and anatomical constraints of the shoulder joint before entry, rather than simply a coordinate drift.
[0050] In step 5, the assessment results are presented as visual charts via a mobile application. Personalized training adjustments are made based on the patient's home environment (such as furniture layout and available exercise space). Longitudinal tracking data is encrypted and uploaded to the hospital's rehabilitation center, where attending physicians can remotely access the patient's biomechanical safety reports to scientifically assess rehabilitation progress.
[0051] Example 4: In the training of functional reconstruction after anterior cruciate ligament (ACL) surgery for professional athletes, this example places higher demands on the parameter accuracy and real-time performance of the method.
[0052] In step 1, a high-speed infrared camera was added to the visual sensor array, increasing the sampling frequency to over 120 frames per second to capture the instantaneous mechanical properties of explosive movements (such as jump landings and sudden stops and turns). In step 2, the ligament tension response mechanism was refined into a bundle-like structure model, which can calculate the stress on the anteromedial and posterolateral bundles of the ACL at different knee flexion angles.
[0053] In step 3, the node features of the graph neural network are incorporated into the prediction of ground reaction force. Although the visual sensor does not directly measure the force, the model can estimate the magnitude of the ground reaction force and its contribution to the joint torque by analyzing the acceleration of the center of gravity and the instantaneous characteristics of the foot landing. In the physical feasibility verification in step 4, shear force threshold monitoring was added. The system monitors the forward displacement trend of the tibia relative to the femur in real time. Once the simulated displacement value exceeds 3 mm or the relevant shear force reaches the preset damage threshold, the system determines it as a potentially dangerous posture and immediately triggers emergency feedback.
[0054] In step 5, the standard rehabilitation movement template incorporates biomechanical norms from elite athletes. In addition to the three basic indicators, the assessment results also include a kinetic symmetry score. By comparing the impact absorption capacity and muscle activation sequence of the left and right limbs at the moment of landing, the system can accurately assess whether the functional recovery of the affected limb has reached the standard for returning to competition. Longitudinal tracking analysis utilizes deep clustering technology to identify whether athletes exhibit compensatory movement patterns (such as using lower back strength to compensate for knee insufficiency) and develops corrective functional strengthening programs accordingly.
[0055] Example 5: This example describes a specific deployment method for a cloud-based orthopedic rehabilitation movement recognition and assessment system. The orthopedic rehabilitation movement recognition and assessment system connects multiple rehabilitation centers via a network, enabling collaborative processing of massive amounts of biomechanical data.
[0056] In the data acquisition phase of step 1, the system supports the access of various visual sensors of different specifications. To achieve cross-platform data compatibility, the system defines a standardized 3D skeleton data protocol. The 3D skeleton data protocol specifies the indexing order of human body key points, the definition method of the coordinate system, and the timestamp encapsulation format. After initial local compression, the acquired data is sent to the cloud computing cluster through a secure transmission tunnel.
[0057] In step 2, a large-scale biomechanical digital twin model library is maintained in the cloud. This library contains tens of thousands of parameter templates based on anatomical statistics of patients of different races and body types worldwide. When a new patient is connected to the system, the cloud uses an automated parameter recognition algorithm to quickly match the closest digital twin model based on the patient's baseline body measurements (height, limb circumference, weight, etc.).
[0058] In step 3, the cloud computing cluster runs the graph neural network using massively parallel processing technology. To improve inference efficiency, the network structure employs model quantization and pruning techniques, reducing computational overhead while maintaining the accuracy of biomechanical constraints.
[0059] In step 4, pose correction, the system employs a multi-model fusion strategy. In addition to biomechanical-based correction, it incorporates a learning model based on massive amounts of real-world motion samples. When large areas of visual data are missing, the system integrates physical laws with historical motion probability distributions to provide pose restoration results that are both scientifically sound and statistically reasonable.
[0060] In step 5, the comprehensive assessment results are integrated into the electronic medical record system. Longitudinal tracking analysis not only targets individual patients but also allows for population trend analysis of thousands of patients with similar conditions in a privacy-de-sensitized manner. This enables the system to identify the general effectiveness or potential risks of certain rehabilitation programs in specific populations, thus providing large-scale data evidence for revising rehabilitation guidelines.
[0061] Example 6: In the context of orthopedic rehabilitation for hemiplegia caused by stroke, this example details how to use the method of the present invention to perform refined motion quality assessment.
[0062] In step 2, the biomechanical digital twin model specifically incorporates parameters related to neuromuscular control failure. For the hemiplegic side, the model lowers the upper limit of the muscle's active contractile force and increases the resistance coefficient during abnormal spasticity to realistically reflect the limb's biomechanical characteristics after central nervous system damage. In the graph neural network of step 3, node features are enhanced with descriptive features about movement smoothness (Jerk value). By calculating the rate of change of acceleration over time, the system can sensitively detect involuntary tremors or pauses during movement.
[0063] In step 4, the physical feasibility verification focuses on coordinated movement patterns. Stroke patients often exhibit typical malcoordination (such as involuntary shoulder shrugging during arm raising). The physical constraint layer classifies this posture, which does not conform to efficient kinetics, as a low-quality movement. In step 5, the assessment results include a dimension of movement coordination. The standard template not only compares static positions but also includes the temporal phase difference of movements between joints. By calculating the phase difference between the affected and unaffected sides when performing the same movement, the system provides detailed rehabilitation stage suggestions (such as the Brunnstrom staging reference).
[0064] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.
Claims
1. A method for recognizing and assessing orthopedic rehabilitation movements based on image recognition, characterized in that, Includes the following steps: Step 1: Use a non-invasive visual sensor array to acquire multi-view image sequences of the individual undergoing rehabilitation training, and extract three-dimensional spatial coordinate data of key points of the human skeleton based on the multi-view image sequences. Step 2: Establish a parametric biomechanical digital twin model, wherein the parametric biomechanical digital twin model includes skeletal structure, muscle distribution and ligament connection relationship, and is preset with joint range of motion constraints, muscle force line direction constraints and ligament tension response mechanism. Step 3: Input the three-dimensional spatial coordinate data into the node structure of the graph neural network. The node structure represents key parts of human anatomy, and the edges between the node structures represent the biomechanical connection relationships between adjacent parts. Step 4: During the inference process of the graph neural network, the parameterized biomechanical digital twin model is invoked to perform a physical feasibility verification on the current posture. When the result of the physical feasibility verification is that the current posture causes the joint angle to exceed the joint range of motion constraint or triggers the ligament overload state in the ligament tension response mechanism, a posture correction operation is performed. Step 5: Based on the posture data after performing the posture correction operation, calculate the spatiotemporal trajectory similarity between the current movement of the individual undergoing rehabilitation training and the standard rehabilitation movement template, and associate it with the biomechanical rationality score to output a comprehensive evaluation result.
2. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, Step 1, which involves acquiring a multi-view image sequence of the rehabilitation training individual using a non-invasive visual sensor array, includes: placing a first depth camera component directly in front of the rehabilitation training individual to capture the features of the frontal posture and the core interference region. The second depth camera component is positioned to the side of the individual undergoing rehabilitation training, and the angle between the optical axis of the second depth camera component and the optical axis of the first depth camera component is between 45 degrees and 90 degrees, in order to acquire lateral depth information. A third depth camera assembly is positioned above the individual undergoing rehabilitation training to provide a vertical spatial reference by covering the training area from a top-down perspective. The first depth camera component, the second depth camera component, and the third depth camera component are synchronized in time by means of hardware synchronization trigger signal or network time protocol, so that the acquisition frequency of each camera component is not less than 30 frames per second. Background removal, illumination compensation, and depth map denoising are performed on the acquired multi-view image sequences. The coordinates of two-dimensional key points under different viewpoints are mapped to a unified global three-dimensional coordinate system through the principle of multi-view geometric projection, generating three-dimensional spatial coordinate data of human skeleton key points.
3. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, In step 2, the parameterized biomechanical digital twin model is established using a layered modeling strategy, which specifically includes: constructing a bottom-level rigid skeletal chain structure, abstracting the human skeleton into rigid links with fixed length and mass distribution characteristics, connecting the rigid links through kinematic pairs, and setting corresponding degrees of freedom and joint range of motion constraints for the kinematic pairs; A deformable muscle envelope layer is constructed in the middle layer. A geometric mesh with biophysical properties is covered on the outside of the rigid skeletal chain structure at the bottom layer to simulate human muscle groups. Force line direction constraints are defined for each muscle group. The force line direction constraints specify the point of application and direction of the force vector generated by the muscle during contraction or stretching. At the same time, cross-sectional area parameters and contraction stress coefficients are introduced to simulate muscle morphological changes and biomechanical support forces. A dynamic ligament tension feedback layer is constructed, which abstracts the ligament as an elastic unit connecting adjacent rigid body links and sets a nonlinear tension response mechanism. When the change in the spacing between the rigid body links causes the strain of the elastic unit to exceed the preset elastic limit, a tension feedback signal is generated. Data interaction between the bottom rigid skeletal chain structure, the middle deformable muscle envelope layer, and the upper dynamic ligament tension feedback layer is achieved through preset mechanical coupling rules. The pose change of the bottom rigid skeletal chain structure drives the middle deformable muscle envelope layer to deform and pulls the upper dynamic ligament tension feedback layer to generate stress.
4. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, Step 2 also includes a model parameter adaptation step: obtaining the age, gender, and pathological status data of the individual undergoing rehabilitation training, wherein the pathological status data includes clinical imaging reports, body composition analysis data, and medical history records; The bone mineral density coefficient, muscle elastic modulus, and articular cartilage friction coefficient within the parameterized biomechanical digital twin model are automatically adjusted based on the age, gender, and pathological condition data. Specifically, for elderly subjects, the values of the bone mineral density coefficient and the muscle elastic modulus were lowered, while the value of the articular cartilage friction coefficient was increased.
5. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, Step 3, inputting the three-dimensional spatial coordinate data into the node structure of the graph neural network, includes: extracting the feature vector of each node, wherein the feature vector includes the three-dimensional spatial coordinates of the corresponding anatomical key parts at the current moment, and the local motion velocity vector and acceleration vector obtained by performing differential operation on continuous image frame data; The weights of the edges between the node structures are dynamically adjusted based on the biomechanical dependence strength between adjacent joints. When performing a lower limb squat, the weight value of the edge connecting the knee joint node and the ankle joint node is increased. During the inference process of each layer of the graph neural network, the information transmission between nodes follows the biomechanical connection relationship, and the motion state of the distal limb is fed back to the proximal core through the kinetic chain.
6. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, The physical feasibility verification process in step 4 includes: performing joint angular velocity mutation detection, calculating the rate of change of joint angles between adjacent frames, and determining whether the physical acceleration corresponding to the rate of change exceeds the achievable range of human physiological structure. If it exceeds the range, it is determined to be an abnormal posture. Perform a muscle synergistic activation consistency judgment. Based on the muscle force line direction constraint, analyze whether the muscle groups involved in the movement under the current posture conform to the synergistic force logic. If the agonist muscle and the antagonist muscle are detected to generate antagonistic contraction force exceeding the preset threshold at the same time, it is determined to be an abnormal posture with logical contradiction. Perform a ligament strain energy threshold comparison, calculate the simulated strain energy of the ligaments at each part under the current posture, and determine that the simulated strain energy exceeds the preset safety threshold when the value exceeds the threshold.
7. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, The attitude correction operation in step 4 employs an inverse dynamics optimization algorithm, specifically including: anchoring the positions of visible key points whose confidence level is higher than a preset probability threshold, as acquired by the non-invasive visual sensor array; For anatomical sites that are obscured or misidentified, the optimal posture solution is sought within the feasible solution space by using the joint range of motion constraints and muscle force line direction constraints in the parameterized biomechanical digital twin model as hard constraints. The optimal posture solution satisfies that all joint angles are within a preset physiological range and the biomechanical potential energy of the whole body is in a local minimum state; the hidden state inside the graph neural network is updated using the coordinate values corresponding to the optimal posture solution to eliminate data collapse at the visual recognition level.
8. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, The process of outputting the comprehensive evaluation result in step 5 includes: finding the matching path with the smallest cumulative distance between the movement sequence of the rehabilitation training individual and the standard rehabilitation movement template sequence, calculating the spatial difference between the two after time axis alignment, so as to determine the spatiotemporal trajectory similarity. The standard rehabilitation exercise template includes a preset sequence of ideal joint angles, the trajectory of the center of gravity projection area on the support surface, and parameters of the ideal muscle exertion pattern for each moment. A biomechanical rationality score is calculated, which reflects the degree of deviation between the actual joint load, predicted muscle fatigue, and energy consumption rate of the individual undergoing rehabilitation training and medical standards during the execution of movements. The comprehensive evaluation results are generated, including a percentage of motion completion reflecting the proportion of the actual trajectory covering the standard trajectory, a biomechanical safety level index based on the degree of joint load deviation from the normal physiological range, and a potential damage risk warning index for accurately identifying sites subjected to improper shear force or excessive pressure.
9. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, The method also includes a longitudinal tracking analysis step: storing multiple rehabilitation training records of the individual undergoing rehabilitation training, and processing the feature vectors generated from each training session using a clustering algorithm; Identify the evolution trend of movement patterns of the individual undergoing rehabilitation training, including trends in changes in maximum joint flexion angles or trends in improvement in gait symmetry; Based on the aforementioned evolution trend, personalized training adjustment suggestions are automatically generated, including increasing the intensity of strength training for specific muscle groups or adjusting the number of repetitions of movements.
10. The orthopedic rehabilitation movement recognition and assessment method based on image recognition according to claim 1, characterized in that, When the method is applied to scoliosis correction training monitoring, the feature is that: in step 2, the skeletal structure is refined into a linkage model containing 24 vertebrae and the sacrum, and each vertebra is connected by a flexible joint with 6 degrees of freedom to simulate the physical characteristics of the intervertebral disc. In step 4, a center of gravity balance constraint is introduced, the position of the human body's center of mass in the current posture is calculated and projected onto the support surface. If the center of gravity projection exceeds the edge of the support area formed by the two feet and the muscles cannot provide sufficient anti-overturning moment, it is determined to be an unsteady artifact and triggers posture correction based on balance mechanics, redistributing the distribution of the body's center of gravity. In step 5, a comparison of the spinal curvature parameter dimension is added to the standard rehabilitation movement template, and the biomechanical safety level is refined into a predicted value of intervertebral pressure.