A markerless motion capture system for patients, computer storage medium

Abstract

This invention belongs to the field of motion capture technology, specifically relating to a markerless motion capture system for patients and a computer storage medium. This invention provides a deep learning-based markerless motion capture system for patients with structural deformities or abnormal gait. The system consists of a synchronization and calibration module, a pose estimation module, a prediction module, a reporting module, and a privacy and data management module. It achieves high-precision, non-invasive 3D motion capture without the need for attached markers. This system eliminates the need for attached markers, reducing interference with patients and exhibiting high adaptability to various factors such as the acquisition site, the number of cameras, and the subject's clothing. By combining multi-view geometry with deep learning, it achieves clinically accurate 3D pose reconstruction with relatively low hardware costs. The localized inference and anonymized data storage technology effectively reduces the risk of patient privacy leaks, making it suitable for various application scenarios such as hospital outpatient clinics and rehabilitation centers.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of motion capture, and particularly relates to a markerless motion capture system for patients and a computer storage medium. BACKGROUND

[0002] Motion capture (MoCap) is a technology for recording and analyzing the movement of moving objects, especially the human body. Its core is to collect motion data through sensors or imaging devices and convert it into a digital form that can be processed, thereby supporting subsequent analysis, visualization and application. Motion capture technology was first applied to film special effects and game character animation, and is also widely used in fields such as sports science (such as athlete technical analysis), rehabilitation medicine (such as gait assessment), virtual reality (VR) and augmented reality (AR).

[0003] According to the sensor type and data acquisition method, existing motion capture systems are mainly divided into marker-based systems and markerless systems. In the marker-based motion capture system, commercial systems such as Vicon and Qualisys are usually equipped with 8 to 20 high-frame-rate infrared cameras, and multiple marker points are arranged at specific anatomical positions of the human body. It can achieve millimeter-level three-dimensional position accuracy and sampling rate higher than 100 Hz, but has the following obvious shortcomings: the hardware equipment is expensive, and each time before collection, it needs to be accurately installed and calibrated by personnel with professional training; the collection site requires strict conditions, such as fixed lighting conditions and unobstructed arrangement space; the marker points are time-consuming to paste and affect the natural performance of the motion, and are easily lost, shifted or blocked in dynamic motion, causing trajectory loss or measurement error; the operation of the supporting data processing software is complex, the learning cost is high, and this type of system is limited in popularization in non-research environments such as hospitals and rehabilitation centers.

[0004] Markerless systems based on inertial measurement units (IMU), such as the Xsens MVN system, have advantages in terms of hardware portability and low requirements for the collection environment. However, their defects are also quite obvious: IMU sensors have integral drift, and the error increases with time accumulation, especially in long-term collection or intense motion, the drift problem is particularly serious; the magnetometer is sensitive to changes in the environmental magnetic field and is easily affected by metal objects or electromagnetic interference, leading to attitude calculation errors; in addition, IMU nodes (usually more than 17) need to be attached to multiple positions on the body, and the wearing process is complicated, and improper wearing can also affect the measurement accuracy. Since the skeletal pose output by the IMU system needs to go through complex Kalman filtering or complementary filtering fusion algorithms, the calculation amount is large, the real-time performance is limited, and the price of commercial IMU systems is still high, making it difficult to deploy on a large scale in primary medical and rehabilitation institutions.

[0005] Gait analysis is an important application of motion capture in the field of medicine and rehabilitation. By recording and analyzing the kinematic data of the lower limbs during walking (such as joint angles, stride length, stride frequency, and support phase ratio), the gait pattern of individuals can be evaluated, abnormalities can be found, and rehabilitation training can be guided. Currently, clinical gait analysis mainly relies on the above-mentioned marker-based optical systems or systems based on inertial sensors, so it inherits their shortcomings in terms of cost, operational complexity, dependence on the environment, and invasiveness to the subject. These limitations are particularly evident in the evaluation of patients with osteoarthritis (OA), as OA patients often have slow and variable gait, and long-term complex device wear or fixed-site collection can affect their natural gait and even cause data distortion.

[0006] In recent years, human pose estimation technology based on deep learning has provided a new approach to markerless motion capture. Such technology uses deep models such as convolutional neural networks (CNNs) and visual Transformers (ViTs) to automatically identify key points (such as hips, knees, ankles, shoulders, etc.) on the human body from single-frame images or video sequences, and to obtain two-dimensional or three-dimensional skeletons through inference. Typical methods include OpenPose, HRNet, VideoPose3D, etc. The core process is usually as follows: collect a large amount of human image or video data with accurate annotations; use deep neural networks to regress the heat map or probability distribution of each key point; find the peak of the heat map to obtain the coordinates of the key points, and connect these key points to form a skeleton. The advantage of deep learning pose estimation is that it does not require the wearing of sensors, can collect data in natural scenes, has strong robustness to lighting and background, and can achieve pixel-level accuracy on public datasets. However, this technology still has limitations: large-scale high-quality labeled data is required for training, the generalization ability under different shooting conditions is limited, and the accuracy will decrease significantly when there is severe occlusion, key points are hidden, or the target motion pattern differs greatly from the training set.

[0007] Existing markerless multi-view RGB methods use two or more synchronized RGB cameras to simultaneously capture from different angles and perform three-dimensional reconstruction through multi-view geometry, combining deep learning to improve the robustness of key point detection and reconstruction. Compared to monocular methods, this method can recover more depth information and has lower cost than traditional marker-based systems. However, this method still has the following shortcomings: (1) the influence of occlusion / clothing / appliances still exists, multi-view can alleviate some occlusions, but may still produce systematic errors when there is severe anatomical deformation or wearing orthotics; (2) training data is limited, even with multi-view methods, public datasets are mostly from healthy individuals or controlled scenarios, lacking large-scale, clinically labeled patient multi-view data, making it difficult for the model to adapt to pathological gait.

[0008] OpenCap is an open-source multi-view markerless motion capture and human pose estimation framework aiming to provide an efficient and flexible solution for human motion capture and analysis. The hardware configuration of the system includes two or three iOS devices (such as iPhone or iPad), a tripod, a checkerboard target for camera calibration, and a computing device running the OpenCap web application. OpenCap can guide users through multi-view camera calibration, real-time acquisition of human motion data, two-dimensional human pose estimation using deep learning models, and reconstruction of three-dimensional skeleton trajectories using inverse kinematics (IK) algorithms. OpenCap has certain advantages in terms of hardware cost and ease of use, but its design target is for general human pose estimation, not for patient gait analysis. In terms of dataset construction, OpenCap's marker augmentation model is based on data from healthy populations, lacking coverage and targeted optimization for patient populations, so when pathological gait features such as gait asymmetry, abnormal pelvic rotation, and knee valgus are involved, the detection accuracy and robustness of the model are insufficient. In addition, OpenCap's data processing and analysis rely on cloud servers, and all video and pose data need to be uploaded to remote platforms for computation, which poses a risk of patient privacy leakage in medical and rehabilitation scenarios, and also limits the application in environments without external networks or strict data security requirements.

[0009] Therefore, it is a problem in the field to construct a high-precision markerless motion capture system for patient populations with structural deformities or abnormal gait. SUMMARY

[0010] To overcome the defects of the prior art, the present application provides a markerless motion capture system for patient populations and a computer storage medium.

[0011] The present application provides a markerless motion capture system for patients, which includes the following modules:

[0012] The synchronization and calibration module is configured to synchronously capture multi-view video streams of the subject by the cameras, and to obtain the intrinsic and extrinsic parameters of the cameras by Zhang's calibration method;

[0013] The pose estimation module is configured to input the video frame images synchronously captured by the cameras into a two-dimensional pose estimation model, to obtain a probability heat map, to output the two-dimensional coordinates and confidence scores of the key points of the lower limbs of the subject in the images, and to input the two-dimensional coordinates, confidence scores or probability heat map in the multi-view images and the camera projection matrix into a three-dimensional pose estimation model to obtain three-dimensional lower limb skeleton time series data; the camera projection matrix is the product of the intrinsic and extrinsic parameters matrices;

[0014] The prediction module is configured to input the three-dimensional lower limb skeleton timing data into a kinematics and dynamics coupled multi-task prediction network, and output a curve of a three-dimensional ground reaction force vector changing over time and a clinical evaluation index.

[0015] The kinematics and dynamics coupled multi-task prediction network adopts an architecture combining a graph convolution network and a Transformer, and is obtained through optimization based on a biomechanics constraint multi-task loss function.

[0016] The formula of the biomechanics constraint multi-task loss function is as follows:

[0017]

[0018] Wherein, L total The biomechanics constraint multi-task loss function is represented as L, L primary The sum of the standard loss functions for each independent prediction task is represented as L, L biomechanics The coupling constraint based on physical laws and clinical knowledge is represented as L, λ bio L is a hyperparameter for balancing the weight between basic task learning and biomechanics constraint.

[0019] Preferably, in the synchronization and calibration module, the number of cameras is 2-8, the frame rate of the camera is ≥60 fps, the resolution is ≥1920×1080, the camera lens is directed towards the capture area, and the angle between each camera is kept at 45°-90°.

[0020] And / or, the camera synchronization process adopts a hardware trigger or a network precise time protocol.

[0021] And / or, the process of collecting multi-view video streams of the subject adopts a checkerboard method.

[0022] Preferably, in the pose estimation module, the network structure of the two-dimensional pose estimation model includes a key point detection model based on a visual Transformer, and the set of key points includes the center of the hip joint, the center of the knee joint, the center of the ankle joint, the head of the first metatarsal bone, the head of the fifth metatarsal bone, the heel, the anterior superior iliac spine of the pelvis, and the posterior superior iliac spine.

[0023] Preferably, in the pose estimation module, the three-dimensional pose estimation model includes a geometric triangulation method and a constraint optimization strategy; the constraint optimization strategy includes an uncertainty-driven robust multi-view fusion algorithm.

[0024] Preferably, in the pose estimation module, the uncertainty-driven robust multi-view fusion algorithm comprises: multiplying the reprojection error term of each view by its corresponding two-dimensional detection confidence score; and taking the variance or entropy of the probability heat map obtained by the two-dimensional pose estimation model as a direct measure of uncertainty.

[0025] Preferably, the constraint optimization strategy further comprises introducing bone segment length constraints, joint angle range limits, foot bottom contact consistency or timing smoothing prior knowledge on the basis of the initial three-dimensional coordinates for global optimization.

[0026] Preferably, in the prediction module, the clinical evaluation indicators include: hip-knee-ankle angle, lower limb length difference, KL classification of knee osteoarthritis patients.

[0027] L primary The specific form of is: ,

[0028] wherein, w 1, w 2, w 3 are weight coefficients of respective tasks, L GRF represents the ground reaction force prediction loss, L HKA represents the hip-knee-ankle angle prediction loss, L KL represents the knee osteoarthritis KL classification prediction loss.

[0029] Preferably, in the prediction module, L biomechanics The specific form of is:

[0030] ,

[0031] wherein, α , β are weight coefficients of respective constraint terms, L kd is the ground reaction force directly predicted by the prediction model GRF pred , t is the mean square error between the force F calc , t calculated according to kinematics, is the difference between the direct prediction value of the prediction model for a certain indicator and the expected value calculated by the consistency function g (⋅).

[0032] Preferably, The specific form of is:

[0033] ,

[0034] Where T represents the total number of time steps, and t represents the time index from 1 to T. Indicates at time step t The ground reaction force predicted by the time prediction model, Indicates at time step t The actual ground reaction force measured at the time;

[0035] And / or, L HKA The specific form is as follows: ,

[0036] in, Indicates at time step t The hip-knee-ankle angle predicted by the prediction model. Indicates at time step t The actual hip-knee-ankle angle measured at the time;

[0037] And / or, L KL The specific form is: ,

[0038] Where C represents the total number of categories in the KL classification, and c represents the category index variable from 1 to C. This represents the true value of the c-th KL classification category. This represents the predicted value for the c-th KL classification category.

[0039] Preferred, The specific form is as follows:

[0040] ,

[0041] Where T represents the total number of time steps, and t represents the time index from 1 to T. Indicates at time step t The ground reaction force predicted by the time prediction model, This represents the equivalent ground reaction force calculated from kinematics.

[0042] And / or, The specific form is: ,

[0043] in, The weight parameters represent the KL classification loss. This indicates the predicted value of the prediction model for the KL classification. This represents the expected value of the KL grade calculated from the predicted hip-knee-ankle angle and the predicted knee adduction moment. The weighted parameters representing hip-knee-ankle angle injuries, This represents the hip-knee-ankle angle predicted by the prediction model. This represents the expected value of the hip-knee-ankle angle calculated from the predicted KL grade and the predicted knee adduction moment.

[0044] Preferably, the patient-oriented markerless motion capture system further includes a reporting module and a privacy and data management module;

[0045] The reporting module is configured to calculate the lower limb hip, knee, and ankle joint moments using inverse dynamics methods based on the ground reaction force predicted by the prediction module; and to determine the gait cycle using a kinematic gait event detection method based on the high-precision three-dimensional skeletal key point temporal data output by the posture estimation module, including calculating gait spatiotemporal parameters such as stride speed, stride length, stride width, stride frequency, gait cycle time, stance phase time, and double stance phase time, and calculating kinematic parameters such as joint angles and range of motion of the hip, knee, ankle, trunk, shoulder, elbow, and / or wrist joints during the gait cycle.

[0046] The privacy and data management module is configured such that data processing takes place on the local computer, and only anonymized skeleton data and gait statistics are retained for long-term storage, while the original video is processed according to the privacy policy.

[0047] The present invention provides a computer storage medium storing a computer program, which, when executed by a processor, implements the patient-oriented markerless motion capture system described in any of the preceding claims.

[0048] This invention addresses the shortcomings of existing motion capture systems in patient gait analysis, including insufficient applicability, high equipment cost, complex operation, and inadequate privacy protection. It provides a deep learning-based, label-free motion capture and gait analysis system for patients. The system primarily consists of a synchronization and calibration module, a pose estimation module, a prediction module, a reporting module, and a privacy and data management module. Specifically designed for the clinical assessment needs of patients with structural deformities or abnormal gaits, it incorporates targeted designs in hardware structure, synchronization and calibration methods, data processing workflows, and algorithm optimization strategies to achieve high-precision, non-invasive 3D motion capture without the need for markers. This system exhibits high adaptability in terms of acquisition location, number of cameras, and subject attire, eliminating the need for markers and reducing patient interference. Employing a combination of multi-view geometry and deep learning, it achieves clinically accurate 3D pose reconstruction with relatively low hardware costs. The localized inference and anonymized data storage technologies effectively reduce the risk of patient privacy breaches, making it suitable for various application scenarios such as hospital outpatient clinics and rehabilitation centers, with promising application prospects.

[0049] Compared with traditional marked motion capture systems, this invention does not require the application of reflective markers to the lower limbs, does not interfere with the subject's natural gait, and avoids the risk of data loss or errors due to markers falling off, shifting, or being obscured. This significantly reduces the professional skill requirements for operators and saves time and technical costs associated with setup and data collection.

[0050] In terms of hardware configuration, this invention has low requirements for camera performance, requiring only a minimum of two ordinary RGB cameras to complete lower limb motion capture. The cameras can be arranged at different angles for multiple perspectives, with fewer restrictions on specific installation locations and layouts. This allows for flexible adaptation to data collection sites of different sizes and shapes, making it particularly suitable for space-constrained environments such as clinical clinics, rehabilitation departments, and research laboratories, thereby further reducing equipment investment and site modification costs. As the number of cameras increases, the accuracy and robustness of 3D reconstruction can be further improved.

[0051] Regarding algorithms and datasets, the deep learning model used in this invention is trained on specially collected gait data from various patients, with gait data acquired through a high-precision labeled system serving as the gold standard. This dataset covers patients of different disease types, ages, genders, body types, and disease stages, featuring a large volume of data and high annotation accuracy. This results in higher accuracy and generalization ability for the model in pathological gait patterns (such as limping, limited knee joint movement, and shortened stride). Compared to existing open-source solutions (such as OpenCap), this invention achieves higher pose reconstruction accuracy and more stable results in patient scenarios.

[0052] This invention eliminates the need for a force measuring platform, predicting ground reaction forces with high accuracy using markerless attitude estimation results. Furthermore, it can predict clinical indicators such as the HKA angle, lower limb length difference, and KL grading, providing valuable reference for clinical decision-making.

[0053] Furthermore, this invention deploys the deep learning model and data processing flow entirely on a local computer, avoiding the external network transmission of raw video and pose data, thus meeting the requirements for medical data privacy and security. It is especially suitable for clinical applications and research environments that require strict protection of patient privacy.

[0054] Obviously, based on the above description of the present invention, and according to common technical knowledge and conventional methods in the field, various other modifications, substitutions or alterations can be made without departing from the basic technical concept of the present invention.

[0055] The following detailed embodiments further illustrate the above-described content of the present invention. However, this should not be construed as limiting the scope of the present invention to the following examples. All technologies implemented based on the above-described content of the present invention fall within the scope of the present invention. Attached Figure Description

[0056] Figure 1 The diagram shows the lower limb hip joint angle and the angle difference between the markerless motion capture system and the traditional marker-based motion capture system of this invention. Figure 1 A shows the results of the sagittal angle of the hip joint versus the gait cycle. Figure 1 B is a graph showing the difference in the sagittal angle of the hip joint measured by the two systems as a function of the gait cycle. Figure 1 C is a graph showing the result of the hip joint coronal plane angle versus the gait cycle. Figure 1 D is a graph showing the difference in the angle of the coronal plane of the hip joint measured by the two systems as a function of the gait cycle. Figure 1 E is a graph showing the angle of the hip joint in the horizontal plane as a function of the gait cycle. Figure 1 F is a graph showing the difference in hip joint angles measured by the two systems over the gait cycle. RMSE represents the root mean square error of the joint angle difference between the two systems.

[0057] Figure 2 The diagram shows the lower limb knee joint angle and the angle difference between the markerless motion capture system and the traditional marker-based motion capture system of this invention. Figure 2 A shows the results of the sagittal angle of the knee joint versus the gait cycle. Figure 2 B is a graph showing the difference in the sagittal angle of the knee joint measured by the two systems as a function of the gait cycle. Figure 2 C is a graph showing the result of the knee joint coronal plane angle versus the gait cycle. Figure 2 D is a graph showing the difference in the angle of the coronal plane of the knee joint as measured by the two systems over the gait cycle. Figure 2 E is a graph showing the angle of the knee joint in the horizontal plane as a function of the gait cycle. Figure 2 F is a graph showing the difference in knee joint angles on the horizontal plane measured by the two systems as a function of the gait cycle. RMSE represents the root mean square error of the joint angle difference between the two systems.

[0058] Figure 3 The diagram shows the lower limb ankle joint angle and the angle difference between the markerless motion capture system of this invention and the traditional marker-based motion capture system. Figure 3 A shows the results of the sagittal angle of the ankle joint as a function of the gait cycle. Figure 3 B is a graph showing the difference in the sagittal angle of the ankle joint measured by the two systems as a function of the gait cycle. Figure 3 C is a graph showing the results of the ankle joint coronal angle as a function of the gait cycle. Figure 3 D is a graph showing the difference in the ankle joint coronal plane angle measured by the two systems as a function of the gait cycle. Figure 3 E is a graph showing the angle of the ankle joint in the horizontal plane as a function of the gait cycle. Figure 3F is a graph showing the difference in ankle joint angles measured by the two systems over the gait cycle. RMSE represents the root mean square error of the joint angle difference between the two systems. Detailed Implementation

[0059] In the following embodiments and experimental examples, the algorithms for data acquisition, transmission, storage, and processing steps not specifically described, as well as the hardware structures and circuit connections not specifically described, can all be implemented using the content already disclosed in the prior art.

[0060] Example 1: A markerless motion capture system for patients

[0061] 1. Hardware Components

[0062] (1) Motion capture area: The recommended length of the area is 4-8 meters to ensure that the patient can complete at least 2 complete gait cycles of walking back and forth. The background should be as simple as possible, with uniform lighting, and strong reflections and shadows should be avoided.

[0063] (2) Image acquisition equipment: at least two RGB cameras (2-8 cameras are used by default, and 4 cameras are used in this embodiment); the camera frame rate is ≥60 fps, the resolution is ≥1920×1080, and the camera is equipped with a fixed focus lens; the camera is fixed on a tripod or stable support, the lens is facing the capture area, and the cameras are kept at an angle of 45°-90° to form sufficient parallax; the camera is connected to the computer through a wired video interface to ensure data transmission bandwidth and stability.

[0064] (3) Computer and graphics processing unit (GPU): used to receive camera video streams, perform data processing and model inference; GPU is used to accelerate deep learning inference and 3D optimization calculation, with a video memory capacity of ≥6GB to support multi-channel video parallel processing.

[0065] 2. Module Composition

[0066] The system in this embodiment integrates a synchronization and calibration module, an attitude estimation module, a prediction module, a reporting module, and a privacy and data management module.

[0067] (1) Synchronization and Calibration Module

[0068] The synchronization and calibration module is configured so that as the patient walks back and forth within the designated area, the cameras simultaneously acquire multi-view video streams and transmit them to the computer. Specifically, hardware triggering (Trigger signal synchronization shutter) ensures inter-frame synchronization of the cameras (using a software instruction-based synchronization scheme, where the control program collaboratively triggers the acquisition of data from four cameras, identifies the host system timestamp within the image frame, and achieves millisecond-level synchronous acquisition of multi-view data); a checkerboard pattern is used to acquire multi-view calibration images of the subject, and the camera's intrinsic and extrinsic parameters are determined using Zhang's calibration method.

[0069] The intrinsic parameters are the camera's focal length (in pixels), principal point coordinates (coordinates of the intersection of the optical axis and the image plane in the image coordinate system), and distortion coefficients (such as radial and tangential distortion coefficients). The extrinsic parameters are parameters describing the camera's position and orientation relative to the calibration template, including rotation matrices and translation vectors.

[0070] In other embodiments, Network Precise Time Protocol (PTP) synchronization can also be selected to ensure inter-frame synchronization.

[0071] (2) Attitude estimation module

[0072] The pose estimation module is configured to input video frame images synchronously acquired by multiple cameras into a two-dimensional pose estimation model, perform lower limb keypoint detection on each video frame, obtain a probability heatmap, and output the two-dimensional coordinates (x, y) and confidence scores of the keypoints in the image. The coordinates of the two-dimensional keypoints from multiple perspectives, the corresponding confidence scores / heatmaps, and the camera projection matrix (a matrix that integrates the intrinsic and extrinsic parameters of the camera, specifically the product of the intrinsic and extrinsic parameter matrices) are input into the three-dimensional pose estimation model. After geometric triangulation and constraint optimization, stable and accurate three-dimensional lower limb skeleton temporal data are obtained, and the three-dimensional coordinates (X, Y, Z) of all lower limb keypoints in each frame are output.

[0073] The network structure of the two-dimensional pose estimation model is a keypoint detection model based on the Visual Transformer (ViT). The keypoint set is defined based on the lower limb anatomical features of osteoarthritis (OA) patients, including keypoints such as the center of the hip joint, the center of the knee joint, the center of the ankle joint, the first metatarsal head, the fifth metatarsal head, the heel, the anterior superior iliac spine, and the posterior superior iliac spine. The output of the two-dimensional pose estimation model is the two-dimensional coordinates (x, y) of each keypoint in the image and its confidence score.

[0074] The 3D pose estimation model uses the standard multi-view geometric triangulation method to calculate the 2D coordinates from different viewpoints.

[0075] The specific constraint optimization strategy is as follows:

[0076] (2.1) Optimization strategy of uncertainty-driven robust multi-view fusion algorithm

[0077] (2.1.1) Minimizing Confidence-Weighted Reprojection Error: The objective function is not simply to minimize the sum of reprojection errors from all 3D points to the 2D image across all views. Instead, the reprojection error term for each view is multiplied by its corresponding 2D detection confidence score. The formula is illustrated below:

[0078] Optimized_3D_Point =argmin Σ (confidence_i || Reprojection(3D_Point,Camera_i)-2D_Point_i ||²). This means that 2D detection points judged as high-confidence (i.e., clear and unobstructed) by the ViT model will have a higher "voice" in the optimization process, while the impact of low-confidence points (which may be occluded or blurred) on the final result will be significantly suppressed.

[0079] (2.1.2) Uncertainty Measurement Based on Heatmap Distribution: As a more refined implementation, the algorithm can directly utilize the probability heatmap output by the 2D pose estimation model. A high-quality detection will produce a heatmap with sharp peaks and a concentrated distribution; while an uncertain detection corresponds to a flat and dispersed heatmap. This algorithm uses the variance or entropy of the heatmap as a direct measure of uncertainty and introduces it into the optimization process. When fusing multi-view information, the system will prioritize trusting view data from "sharp" heatmaps. Through this adaptive weighted fusion mechanism, the system can actively identify and "filter" out unreliable 2D observation data, thus outputting a stable and anatomically consistent 3D skeleton even in challenging scenarios with severe occlusion and abnormal poses, greatly improving the accuracy and robustness of pose estimation for pathological gait.

[0080] (2.2) Other optimization strategies

[0081] Based on the initial three-dimensional coordinates, various priors such as bone segment length constraints, joint range limitations, plantar contact consistency, and temporal smoothing are introduced for global optimization.

[0082] Traditional multi-view reconstruction algorithms assign equal weight to observations from all views. When keypoints in a particular view are inaccurately detected due to occlusion, it directly introduces huge errors, contaminating the final 3D reconstruction result. The optimization strategy in this embodiment can solve the problem of frequent self-occlusion and atypical postures caused by patients' pathological gait (such as limping, ankylosing knee, compensatory gait, etc.).

[0083] (3) Prediction module

[0084] The prediction module is configured to input high-precision 3D skeletal keypoint temporal data (e.g., a sequence of consecutive frames within a complete gait cycle) output by the pose estimation module into a coupled kinematics-kinetics multi-task network. This multi-task network employs an architecture combining graph convolutional networks (GCNs) and transformers. The GCN layers are responsible for capturing the spatial structural relationships of the human skeleton as a graph within each frame (e.g., the linkage between the hip, knee, and ankle joints), while the Transformer layers are responsible for learning the temporal dependencies within the gait cycle. Multiple parallel output heads are connected to the network backbone, each used to predict different target tasks, namely:

[0085] Ground reaction force (GRF) prediction head: Outputs a curve showing the change of a three-dimensional GRF vector over time.

[0086] Clinical indicator prediction head: output of static or dynamic clinical assessment indicators such as hip-knee-ankle angle (HKA), bilateral lower limb length difference, and KL grade of patients with knee osteoarthritis.

[0087] This multi-task prediction network is trained using a biomechanically constrained multi-task loss function. The following are the details of the biomechanically constrained multi-task loss function:

[0088] (3.1) Biomechanically constrained multi-task loss function: It is not simply the sum of the loss functions of each task (such as mean squared error or cross-entropy). Instead, it is composed of two weighted parts: PrimaryTask Loss. L primary Biomechanical Constraint Loss L biomechanics The details are as follows:

[0089] L total = L primary + λ bio L biomechanics

[0090] in:

[0091] L primary It is the sum of standard loss functions for each independent prediction task, used to drive the model to learn the basic mapping from the input (3D pose sequence) to each output metric.

[0092] L biomechanics It introduces coupled constraints based on physical laws and clinical knowledge to penalize predictions that do not conform to biomechanical principles.

[0093] λ bio It is a hyperparameter used to balance the weights between basic task learning and biomechanical constraints.

[0094] (3.2) L primary

[0095] L primary It is a weighted sum of the losses of multiple subtasks, specifically in the form of:

[0096] L primary = w 1 L GRF + w 2 L HKA + w 3 L KL

[0097] in w i These are the weighting coefficients for each task. The loss function for each subtask is selected based on its task type:

[0098] Ground reaction force (GRF) predicts loss L GRF The loss is calculated using Mean Squared Error (MSE).

[0099] 2

[0100] HKA angle and other clinical indicators loss L HKA For continuous indicators such as HKA angle and lower limb length difference, mean squared error (MSE) loss is used:

[0101] 2

[0102] KL classification indicators and other classification indicators loss L KL : Using cross-entropy loss:

[0103]

[0104] (3.3) L biomechanics

[0105] L biomechanics It forces the model to learn physical and physiological connections beyond the surface of the data. It consists of two key coupling losses:

[0106]

[0107] in α and β These are the weighting coefficients of their respective constraint terms.

[0108] (3.3.1) Kinematic-dynamic coupling loss L kd (Kinematics-Dynamics Coupling Loss)

[0109] This loss term is based on Newton's second law ( F = ma This establishes a physical connection between lower limb kinematics (posture changes) and dynamics (ground reaction force). The specific implementation method is as follows:

[0110] (3.3.1.1) Calculate the acceleration of the center of mass (CoM) of the lower limb:

[0111] First, based on the time sequence of 3D human body key points output by the model... P 3D ( t Using standard anthropometry models (such as the Dempster model), the mass of each segment of the lower limb (thigh, calf, foot) is calculated, and the sequence of the center of mass positions of the entire lower limb system is further calculated. CoM pos ( t Then, by analyzing the centroid position sequence... CoM pos ( t By performing a second numerical differential, the acceleration sequence of the lower limb's center of mass is obtained. CoM ( t ).

[0112] (3.3.1.2) Calculate the ground reaction force derived from kinematics:

[0113] According to Newton's second law, the equivalent ground reaction force calculated from kinematics should be:

[0114]

[0115] in m lower_limb It is the total mass of the lower limbs. G lower_limb It is the force of gravity acting on the lower limbs.

[0116] (3.3.1.3) Define the loss function L kd

[0117] L kd Defined as the ground reaction force directly predicted by the model. GRF pred ( t ) and the force calculated from kinematics F calc ( t The mean square error between )

[0118] 2

[0119] By minimizing L kd The model is "forced" to learn the intrinsic physical relationship between kinematics and dynamics, thereby avoiding physically inconsistent results such as "the posture shows that the body is accelerating downwards, but the GRF prediction is much greater than gravity", thus improving the robustness and realism of the prediction.

[0120] (3.3.2) Loss of correlation between clinical indicators L ci (Clinical Indicators Correlation Loss)

[0121] Different clinical indicators are not independent of each other, but rather have an intrinsic relationship that has been confirmed by clinical research. For example, genu varum (high HKA angle) usually leads to increased load on the medial knee joint, which is reflected in an increase in knee adduction moment (KAM) and is highly correlated with more severe KL classification. L ci The goal is to encode this prior knowledge into the loss function, penalizing prediction combinations that are inconsistent with clinical logic. The specific implementation method is as follows:

[0122] (3.3.2.1) Establish a correlation model between indicators

[0123] First, based on the prediction GRF pred ( t ) and three-dimensional pose P3D ( t This allows for the calculation of derived biomechanical parameters, such as knee adduction torque. KAM pred( t ).

[0124] Then, this embodiment defines one or more consensus functions. g ( This function represents the known relationship between indicators. It is a pre-trained function that acts as a built-in "statistical arbitrator," essentially fixing prior knowledge derived from real clinical data into a mathematical function of a statistical model. This function can be a simple linear model, a lookup table based on clinical statistics, or a small, pre-trained neural network used to simulate, for example, the clinical logic of predicting KL grading from HKA angles and KAM. For example, the function... g kl ( HKA pred , KAM pred The output of ) is the expected KL classification based on the input HKA and KAM.

[0125] With function g kl ( HKA pred , KAM pred For example, given HKA and KAM as input, the output is the expected probability distribution P = [p0, p1, p2, p3, p4] of KL grade (0-4). When g(HKA=5, KAM=0.08) outputs [0, 0.1, 0.2, 0.7, 0], it indicates that, based on the statistical data of the training set, 70% of patients with this HKA and KAM are KL grade 3.

[0126] (3.3.2.2) Define the loss function L ci

[0127] L ci Defined as the direct prediction of a certain indicator by the model, versus the prediction obtained through a consistency function. g ( The difference between the expected values ​​derived from other relevant indicators.

[0128]

[0129] This loss term encourages all output heads of the model (GRF, HKA, KL, etc.) to produce mutually supportive and clinically consistent predictions. For example, when the model predicts a high KL grade, a large loss will occur if its predicted HKA angle and calculated KAM value are not correspondingly within the clinically relevant range.

[0130] (4) Reporting module

[0131] The reporting module is configured to calculate the torques of the hip, knee, and ankle joints of the lower limbs using the inverse dynamics method based on the ground reaction force predicted by the prediction module; and to determine the gait cycle using a kinematic gait event detection method based on the high-precision three-dimensional skeletal key point temporal data output by the posture estimation module. Specifically, this involves calculating gait spatiotemporal parameters: gait speed, stride length, stride width, gait frequency, gait cycle time, stance phase time, and double stance phase time; and calculating kinematic parameters: joint angles and joint range of motion for each joint during the gait cycle (hip, knee, and ankle joints of the lower limbs, trunk, shoulder, elbow, and wrist joints, etc.). The measurement and calculation of joint angles and joint range of motion are preferably performed using sagittal plane measurements and calculations.

[0132] (5) Privacy and Data Management Module

[0133] The privacy and data management module is configured to complete the reasoning and analysis process on the local computer, without relying on external servers; long-term storage retains only anonymized skeleton data and gait statistics, while deleting identifiable information. In other words, results visualization and export are completed locally, and the original video is processed according to the privacy policy.

[0134] The technical solution of the present invention will be further explained through experiments below.

[0135] Experiment Example 1: Lower Limb Motion Capture Experiment for Patients with Structural Deformities or Abnormal Gait

[0136] I. Experimental Methods

[0137] The dataset information for this invention is shown in the table below.

[0138] Table 1 Dataset Information

[0139]

[0140] Using the markerless motion capture system of Embodiment 1 of the present invention and the traditional marked optical motion capture system (Vicon), the kinematic parameters of the lower limbs are predicted and calculated respectively: the joint angles and joint range of motion of each joint during the gait cycle (hip, knee and ankle joints of the lower limbs), and the difference parameters between the two systems are calculated.

[0141] The root mean square error (RMSE) is a commonly used indicator to measure the difference between two systems. It reflects the magnitude of the difference between the measured value and the true value (or reference value). The smaller the RMSE, the smaller the difference between the two measurement methods.

[0142] The difference in joint range of motion refers to the difference between the joint range of motion measured by two systems. A positive value indicates that the range of motion measured by the unmarked system is higher than that measured by the marked system, while a negative value indicates that the range of motion measured by the unmarked system is lower than that measured by the marked system.

[0143] The intragroup correlation coefficient (ICC) is an indicator that measures the consistency and reliability between two systems, reflecting the repeatability and stability of measurement results. Generally, ICC < 0.5 indicates poor consistency; 0.5 ≤ ICC < 0.75 indicates moderate consistency; 0.75 ≤ ICC < 0.9 indicates good consistency; and ICC ≥ 0.9 indicates excellent consistency.

[0144] II. Experimental Results

[0145] The results of the lower limb joint angle test are shown in Table 2 and Figure 1 As shown. Compared with traditional marked systems, the measurement results of the markerless system of this invention have smaller errors and better accuracy. Different anatomical planes have a significant impact on the measurement results. The results in the sagittal plane have high consistency and reliability and can be directly used for clinical decision-making. The consistency of the measurement results in the coronal plane for the hip and ankle joints and the horizontal plane for the ankle joint is moderate and can be used as a reference. This indicates that the markerless system of this invention has high accuracy in skeletal deformities (knee valgus and varus deformities, lower limb length discrepancies, etc.) and pathological gait patterns (such as limping, limited knee joint movement, shortened stride, etc.), and can better replace traditional marked systems for measuring sagittal joint angles.

[0146] Table 2. Lower limb joint angle error parameters (mean ± standard deviation) between the markerless system of this invention and the traditional marked system.

[0147]

[0148] As can be seen from the above embodiments and experimental examples, this invention addresses the shortcomings of existing motion capture systems in patient gait analysis, such as insufficient applicability, high equipment cost, complex operation, and inadequate privacy protection. It provides a deep learning-based label-free motion capture and gait analysis system for patients. This system mainly consists of a synchronization and calibration module, a pose estimation module, a prediction module, a reporting module, and a privacy and data management module. Specifically designed for the clinical assessment needs of patients with structural deformities or abnormal gait, it incorporates targeted designs in hardware structure, synchronization and calibration methods, data processing flow, and algorithm optimization strategies to achieve high-precision, non-invasive 3D motion capture without the need for markers. This system exhibits high adaptability in terms of acquisition site, number of cameras, and subject attire, eliminating the need for markers and reducing patient interference. By combining multi-view geometry with deep learning, it achieves clinically accurate 3D pose reconstruction with low hardware costs. The localized inference and anonymized data storage technologies effectively reduce the risk of patient privacy leakage, making it suitable for various application scenarios such as hospital outpatient clinics and rehabilitation centers, with promising application prospects.

Claims

1. A patient-oriented markerless motion capture system, characterized in that, It comprises the following modules: A synchronization and calibration module is configured to synchronize the cameras to collect multi-view video streams of a subject, and to obtain intrinsic and extrinsic parameters of the cameras by Zhang's calibration method; A pose estimation module is configured to input video frame images synchronously collected by the cameras into a two-dimensional pose estimation model to obtain a probability heat map, and to output two-dimensional coordinates and confidence scores of key points of the subject's lower limbs in the images, and to input the two-dimensional coordinates, confidence scores or probability heat map and camera projection matrix in multi-view images into a three-dimensional pose estimation model to obtain three-dimensional lower limb skeleton time series data; the camera projection matrix is the product of the intrinsic and extrinsic parameters; A prediction module is configured to input the three-dimensional lower limb skeleton time series data into a kinematics and dynamics coupled multi-task prediction network to output a curve of the three-dimensional ground reaction force vector changing over time and a clinical evaluation index; The kinematics and dynamics coupled multi-task prediction network adopts an architecture combining a graph convolution network and a Transformer, and is obtained by optimization based on a biomechanics constraint multi-task loss function; The formula of the biomechanics constraint multi-task loss function is: wherein, L total represents the multi-task loss function based on biomechanical constraints, L primary represents the sum of standard loss functions for each individual prediction task, L biomechanics represents coupling constraints based on physical laws and clinical knowledge, λ bio is a hyper-parameter for balancing the weight between the base task learning and the biomechanical constraints; L primary In particular forms, the application provides: , wherein, w 1、 w 2、 w 3 are weight coefficients of each task, respectively, L GRF denotes the ground reaction force prediction loss, L HKA denotes the hip-knee-ankle angle prediction loss, L KL denotes the knee osteoarthritis KL grade prediction loss; L biomechanics In particular, the application relates to a compound of formula (I) ， wherein, α , β are weight coefficients of the respective constraint terms, L kd is the mean square error between the ground reaction force directly predicted by the prediction model GRF pred ( t ) and the force calculated from kinematics F calc ( t ), is the difference between the direct prediction value of the prediction model for a certain index and the expected value calculated by the consistency function g ( ). In particular, the application relates to a compound of formula (I) , wherein represents a weight parameter for the hip-knee-ankle angle impairment, represents a predicted value of the hip-knee-ankle angle predicted by the prediction model, represents an expected value of the hip-knee-ankle angle derived from the predicted KL classification and the predicted knee-in-toe moment, represents a weight parameter for the KL classification, represents a predicted value of the hip-knee-ankle angle predicted by the prediction model, represents an expected value of the hip-knee-ankle angle derived from the predicted KL classification and the predicted knee-in-toe moment.

2. The patient-facing markerless motion capture system of claim 1, wherein: In the synchronization and calibration module, the number of cameras is 2-8, the frame rate of the cameras is ≥60 fps, the resolution is ≥1920×1080, the camera lens is directed towards the capture area, and the angle between each camera is kept at 45°-90°; And / or, the camera synchronization process adopts a hardware trigger or a network precise time protocol; And / or, the process of collecting multi-view video streams of the subject adopts a checkerboard method.

3. The patient-facing markerless motion capture system of claim 1, wherein: In the pose estimation module, the network structure of the two-dimensional pose estimation model includes a key point detection model based on a visual Transformer, and the set of key points includes the center of the hip joint, the center of the knee joint, the center of the ankle joint, the head of the first metatarsal bone, the head of the fifth metatarsal bone, the heel, the anterior superior iliac spine, and the posterior superior iliac spine.

4. The patient-facing markerless motion capture system of claim 1, wherein: In the pose estimation module, the three-dimensional pose estimation model includes a geometric triangulation method and a constraint optimization strategy; the constraint optimization strategy includes an uncertainty-driven robust multi-view fusion algorithm.

5. The patient-facing markerless motion capture system of claim 4, wherein: In the pose estimation module, the uncertainty-driven robust multi-view fusion algorithm includes: the reprojection error term of each view is multiplied by its corresponding two-dimensional detection confidence score; the variance or entropy of the probability heat map obtained by the two-dimensional pose estimation model is used as a direct measure of uncertainty; And / or, the constraint optimization strategy further includes introducing bone segment length constraints, joint angle range restrictions, foot bottom contact consistency, or prior knowledge of temporal smoothing for global optimization based on initial three-dimensional coordinates.

6. The patient-facing markerless motion capture system of claim 1, wherein: In the prediction module, the clinical evaluation index includes the hip-knee-ankle angle, the difference in the length of the two lower limbs, and the KL classification of patients with knee osteoarthritis.

7. The patient-facing markerless motion capture system of claim 6, wherein, In particular, the compound of formula (I) is: ， where T denotes the total number of time steps, t denotes a time index from 1 to T, denotes the ground reaction force predicted by the prediction model at time step t t, denotes the ground reaction force actually measured at time step t t. and / or, L HKA In particular, the application relates to a compound of formula (I) , wherein, represents the hip-knee-ankle angle predicted by the prediction model at time step t represents the hip-knee-ankle angle actually measured at time step represents the hip-knee-ankle angle predicted by the prediction model at time step t represents the hip-knee-ankle angle actually measured at time step and / or, L KL In particular, the application relates to a compound of formula (I) , where C denotes the total number of classes in the KL hierarchy, c denotes a class index variable from 1 to C, denotes the true value of the cth KL hierarchy class, denotes the predicted value of the cth KL hierarchy class.

8. The patient-facing markerless motion capture system of claim 6, wherein, In particular, the application relates to a compound of formula (I) ， where T denotes the total number of time steps, t denotes a time index from 1 to T, denotes the ground reaction force predicted by the prediction model at time step t t, denotes the kinematically computed equivalent ground reaction force.

9. The patient-facing markerless motion capture system of claim 1, wherein, The patient-oriented markerless motion capture system further comprises a reporting module and a privacy and data management module; The reporting module is configured to calculate the lower limb hip, knee, and ankle joint torques by inverse dynamics method based on the ground reaction force predicted by the prediction module. The high-precision three-dimensional skeleton key point time series data output by the pose estimation module is determined by a gait event detection method based on kinematics to determine a gait cycle, including calculating gait space-time parameters: step speed, step length, step width, step frequency, gait cycle time, support period time, double support period time, and calculating kinematics parameters: joint angles and joint activities of hip joints, knee joints, ankle joints, torso joints, shoulder joints, elbow joints and / or wrist joints in a gait cycle; The privacy and data management module is configured to perform data processing on a local computer, and only anonymized skeleton data and gait statistics are retained for long-term storage, and original videos are processed according to a privacy policy.

10. A computer storage medium storing a computer program, characterized in that, The computer program, when executed by a processor, implements the markerless motion capture system for a patient according to any one of claims 1-9.

Images