Limb movement assessment method based on uncertainty guide discriminant feature learning
By using uncertainty-guided multi-instance learning and comparative feature optimization of branch structure, the problems of interference information and difficulty in distinguishing scoring levels in limb movement assessment are solved, resulting in more stable and interpretable movement scoring.
Patent Information
- Application Number
- CN202511880293.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-13
- Publication Date
- 2026-04-28
AI Technical Summary
Existing methods for assessing body movements suffer from insufficient ability to identify key time periods due to interference in real-world scenarios, and they also struggle to distinguish subtle differences in rating levels, leading to misjudgments by the model between critical levels.
The model employs uncertainty-guided multi-instance learning and comparative feature optimization to optimize the branch structure. It identifies key action segments through an uncertainty estimation mechanism and corrects fuzzy samples through comparative learning in the feature space, thereby enhancing the model's ability to distinguish between adjacent rating levels.
It effectively suppresses the influence of interfering segments, improves the stability and interpretability of action scores, enhances the differentiation of fuzzy samples, and improves the model's discriminative performance between adjacent score levels.
Smart Images

Figure CN121937933A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a method for evaluating limb movements based on uncertainty-guided discriminative feature learning, belonging to the field of computer vision. Background Technology
[0002] Limb movement assessment is a crucial means of measuring human motor function and movement quality, playing a key role in determining the standardization, coordination, and stability of movements. Quantitative analysis of the temporal patterns, spatial amplitude, and postural changes of movements can provide a scientific basis for the objective measurement of motor ability and the identification of abnormal behaviors. In the field of medical rehabilitation, movement assessment is widely used for detecting functional impairments and monitoring rehabilitation progress in patients with neurological diseases. In sports training and human-computer interaction, movement assessment also helps in posture correction, optimization of motor performance, and feedback regulation of intelligent control systems. However, existing assessment methods largely rely on manual observation or experience-based scoring, which are highly subjective, inefficient, and lack consistency, making it difficult to meet the needs of precise diagnosis and large-scale automated assessment. Therefore, there is an urgent need for an automated and objective method for limb movement assessment.
[0003] Existing research on automated analysis of limb data primarily relies on deep learning methods and video or skeletal sequence data. Specifically, Deb et al. [1] A rehabilitation movement assessment model based on Graph Convolutional Network (GCN) is proposed. By modeling the skeletal sequences of rehabilitation patients, it achieves automated prediction of movement quality scores. This method can capture spatiotemporal correlation information and exhibits high accuracy in rehabilitation scenarios. (Lei et al.) [2] This study utilizes key human body information in videos, combining spatiotemporal attention mechanisms with skeletal features to model and analyze the motion execution process. (Bian et al.) [3] A graph convolutional structure was used to establish a representation of the relationships between action features, and feature alignment and discriminative learning were performed based on feature pairs from different sequences. Deng et al. [4] By utilizing video tracking and skeletal motion modeling, the clinical motor status of Parkinson's disease patients was automatically identified and quantified, providing a reliable and interpretable means for monitoring motor function in neurological diseases.
[0004] It is evident that most existing research revolves around the overall modeling of action recognition and quality assessment, typically using video or skeletal sequences as input and employing deep neural networks to achieve end-to-end prediction from raw data to scoring results. These methods perform well on standardized action datasets or task-specific datasets, but still have the following problems in real-world limb action assessment scenarios: (1) Limb action videos often contain a large number of non-critical time periods and invalid posture segments, such as action preparation, pauses, and other interfering information, resulting in significant noise in temporal features. Moreover, most existing models only pursue global accuracy and lack temporal interpretability and confidence constraint mechanisms for the model's prediction process; (2) The differences in actions between different scoring levels are relatively subtle, while the differences within the same level are large, forming a feature distribution characteristic of "small inter-class differences and large intra-class differences," which affects the model's ability to distinguish between critical scoring levels.
[0005] Based on the aforementioned issues, the task of assessing body movements urgently needs to achieve more refined temporal understanding and feature discrimination capabilities at the model level. On the one hand, the model should be able to automatically identify key frames in complex action videos and reduce the influence of interfering periods, just like human observation, thereby obtaining more representative temporal features. On the other hand, it also needs to more accurately characterize the subtle differences between different rating levels in the feature space to avoid misjudgment of critical samples due to ambiguous distribution. Therefore, existing action assessment methods need further improvement. Summary of the Invention
[0006] Existing methods for assessing body movements based on video data often employ end-to-end deep learning models to uniformly model and score the entire movement sequence. However, real-world body movement data contains numerous invalid or interfering segments, such as movement preparation and pauses. This temporal noise significantly weakens the model's ability to perceive key movement phases. Furthermore, body movement samples exhibit a distribution characteristic in the feature space where there are significant differences between similar levels and high similarity between different levels. This means that the execution of movements of the same severity exhibits strong individual differences, while the boundaries between different severity levels are blurred, leading to a decrease in the model's ability to distinguish between adjacent levels and a high likelihood of misclassifying critical samples. To address these challenges, this invention proposes a body movement assessment method based on uncertainty-guided discriminative feature learning. This method achieves adaptive focusing on key temporal segments and feature correction for ambiguous samples, enabling the model to more accurately distinguish between high-confidence movements and ambiguous segments in complex movement sequences, thereby achieving stable and interpretable movement quality scoring.
[0007] This invention adopts the following technical solution: Under the overall spatiotemporal modeling framework, a dual-branch structure is constructed, consisting of an uncertainty-driven multi-instance learning branch and a fuzzy sample comparative feature optimization branch. The former introduces an uncertainty estimation mechanism to perform confidence modeling and weighted aggregation on multiple instance segments after the action video is segmented, achieving feature fusion of key frames; the latter utilizes the principle of contrastive learning in the feature space to dynamically identify and correct confidence samples and fuzzy samples, enhancing the model's ability to distinguish adjacent rating levels. Confidence samples refer to samples that are correctly predicted by the model and whose category determination is stable (i.e., true positive samples), whose features have good intra-class consistency and can be used as feature prototypes for comparison constraints of each category; fuzzy samples refer to samples that are incorrectly predicted or have unclear boundaries, including false negative samples that should belong to a certain category but are misclassified as other categories, and false positive samples that are incorrectly classified into a certain category. Specifically, the multi-instance learning branch first uses a graph convolutional network to extract the spatiotemporal features of each action segment and calculates the instance-level prediction confidence. Then, it fuses instance features based on confidence weights to obtain the global score. The contrastive feature optimization branch enhances feature discrimination through decoupling spatial and temporal features, class prototype alignment, and fuzzy sample correction. Finally, the system fuses the outputs of the two branches, ensuring the stability and interpretability of the global score while improving the model's discrimination accuracy and robustness between critical classes.
[0008] The specific solution of the present invention is attached. Figure 1 As shown.
[0009] Step 1: Video Preprocessing
[0010] The main task of video preprocessing is to transform the raw patient video data into a standardized skeleton sequence that can be used for graph convolutional modeling, providing stable and uniform data input for subsequent spatiotemporal modeling. The input is the original motion assessment video, and the output is the normalized skeleton sequence.
[0011] Step 1.1 Skeleton Sequence Extraction: The purpose of this step is to extract the spatiotemporal motion information of the human body from the video. From the input motion evaluation video, the pose estimation algorithm is used to extract the human skeleton joint data frame by frame to construct a complete temporal skeleton sequence.
[0012] Step 1.2 Coordinate Normalization: The extracted skeletal sequence undergoes coordinate alignment, denoising, and normalization to eliminate the influence of individual body size differences and camera angles. The input is the original skeleton coordinates, and the output is the normalized coordinates.
[0013] Step 2: Uncertainty-Driven Multi-Instance Learning
[0014] This step aims to adaptively identify and focus on key action segments by introducing an uncertainty assessment mechanism, thereby reducing the impact of invalid or interfering segments on the global score. The input is a normalized skeleton sequence, and the output is a weighted aggregate of global features and the confidence weights of each instance.
[0015] Step 2.1 Instance Partitioning: Divide the long temporal skeleton sequence into several short segments (instances) of fixed or adaptive length. The input is the complete skeleton sequence, and the output is a set of instance segments. The partitioning adopts a sliding window mechanism or a key pose detection algorithm to ensure that each instance segment completely covers a continuous motion phase.
[0016] Step 2.2 Instance Feature Extraction: This step aims to jointly model the spatial structure and temporal dynamics of patient movements using a spatiotemporal graph convolutional network (SPCN) to extract movement feature representations. Each instance is input into the ST-GCN backbone network, and within each frame, spatial neighborhood features are aggregated along the inherent joint topology of the human body. Convolution is performed on the joint features along the temporal dimension to capture their short-term motion patterns, resulting in a spatiotemporal feature map for each instance.
[0017] Step 2.3 Instance Confidence Calculation: Features are independently extracted for each instance segment, and an instance-level score is generated. Simultaneously, the confidence score of the instance is calculated through uncertainty estimation to reflect the reliability of the segment prediction result. The input is the instance feature map, and the output is the instance's score and confidence score. Specifically, after extracting spatial and temporal features through a graph convolutional network, each instance obtains a compact instance representation through a global average pooling layer, and then a linear transformation is applied to map it to the instance's confidence score. This confidence score reflects the model's certainty regarding the segment's prediction result: instances with high confidence typically contain significant action features or key behavioral patterns, while instances with low confidence often correspond to unstable segments such as pose blurring, action pauses, or background interference. Through this uncertainty-based learning modeling approach, the model can adaptively adjust the weights of different instances during training, thereby highlighting reliable segments and weakening the influence of noisy segments during global aggregation, achieving focus on key stages of action and enhanced interpretability.
[0018] Step 2.4 Instance Feature Fusion: Based on the confidence level of each instance, the segment features are weighted and aggregated using an inverse confidence weighting strategy. High-confidence segments contribute more to the global score, thus highlighting key action stages and suppressing interference from non-key segments. The input consists of feature maps and corresponding confidence levels for all instances, and the output is the globally aggregated features.
[0019] Step 3: Comparative Feature Optimization
[0020] This step is an auxiliary module that runs only during training. Its core purpose is to decouple spatial and temporal features and perform prototype-based contrastive correction by embedding a contrastive feature optimization module in the last layer of the backbone network. This enhances the discriminativeness and robustness of the feature representation and improves the model's ability to distinguish within the critical scoring range. The input is the spatiotemporal features of the last layer, and the output is the optimized feature representation (used for loss calculation).
[0021] Step 3.1 Spatial and Temporal Feature Decoupling: The purpose of this step is to separate the different dimensions of information in the spatiotemporal features, enabling the model to specifically optimize spatial topological differences and temporal rhythm differences. The input is the spatiotemporal feature map of the last layer, and the output is the decoupled spatial and temporal features. Specifically, after the multi-layer output of the backbone network, a lightweight feature optimization module is introduced to decouple the hidden spatiotemporal feature map into two branches: spatial and temporal. Specifically, the original features are fed into two parallel channels: the spatial channel performs feature aggregation in the temporal dimension to extract the topological structure and body posture distribution information between joints; the temporal channel performs feature aggregation in the spatial dimension to extract the dynamic changes and rhythmic patterns between frames during the action. Subsequently, each branch performs linear transformation and normalization on the extracted features to obtain spatial and temporal features with uniform dimensions and compact expression.
[0022] Step 3.2 Confidence and Fuzzy Sample Identification: The purpose of this step is to filter out reliable category prototype samples and fuzzy samples that need correction. The inputs are decoupled spatial and temporal features, instance prediction results, and the true labels of the samples. The outputs are a set of confidence samples, a set of fuzzy samples, and category prototypes. Confidence samples and fuzzy samples are dynamically identified based on the model's prediction confidence, and category-level prototype representations are established accordingly. Specifically, samples with correct predictions and high confidence are considered confidence samples and used to update the global prototype centers of each category to stabilize the feature distribution of that category. Simultaneously, samples with incorrect predictions or low confidence are considered fuzzy samples and used to assist in subsequent comparative optimization. Feature correction is achieved through distance constraints with the correct category prototypes. This mechanism can dynamically distinguish between reliable and uncertain samples during training, enabling the network to use robust category prototypes as a reference in comparative learning, thereby improving the consistency and discriminative power of feature representations.
[0023] Step 3.3 Prototype Comparison and Feature Correction: Feature comparison learning is performed using category prototypes as anchors. Confidence samples are aggregated towards their respective category prototypes by maximizing similarity, while ambiguous samples are constrained to move away from erroneous category prototypes, thereby strengthening the inter-class distinction boundary and achieving self-correction at the feature level. The purpose of this step is to optimize the feature space distribution, making similar samples more compact and dissimilar samples more separated. The inputs are decoupled features, category prototypes, and ambiguous samples, and the output is the corrected features.
[0024] Step 3.4 Feature Optimization and Supervision: Integrating the corrected features from the spatial and temporal branches, calculate the contrastive learning loss and jointly optimize it with the main task loss to enhance feature discriminativeness. The purpose of this step is to solidify the feature optimization effect through loss constraints. The inputs are the corrected spatial features, temporal features, and class prototypes, and the output is the contrastive loss.
[0025] Step 4: Joint Optimization and Scoring Output
[0026] This step aims to integrate multi-task losses, achieve end-to-end optimized training, and output the final action score results and interpretability information.
[0027] Step 4.1 Joint Loss Optimization: The purpose of this step is to train the entire model end-to-end through multi-task learning. During the model training phase, end-to-end learning is achieved through a multi-task joint optimization strategy. The overall optimization objective consists of two types of loss terms:
[0028] The main loss is used to constrain the model's basic scoring task to ensure the accuracy of action recognition or score prediction. The contrastive optimization loss draws on the idea of contrastive feature optimization to distinguish and correct fuzzy samples from clear samples, strengthen the consistency of intra-class features and expand the differences between classes, thereby improving the model's ability to distinguish between adjacent rating levels.
[0029] These two parts work together to enable the model to simultaneously improve feature extraction, instance aggregation, and discrimination optimization during end-to-end training, providing robust support for subsequent score prediction and interpretable output.
[0030] Step 4.2 Evaluation Score Prediction: The purpose of this step is to evaluate the new video using the trained model. After the model is trained, the input is the skeletal sequence of the complete video, and the output is the final score of the action, along with the temporal location and weight distribution of key segments, providing quantitative and interpretable auxiliary evaluation data for limb action assessment.
[0031] Compared with existing technologies, the advantages of this invention are as follows:
[0032] I. The limb movement assessment method based on uncertainty-guided discriminative feature learning adopted in this invention can effectively suppress the influence of non-critical and interfering segments, achieve adaptive focusing on key movement stages, and improve the stability and temporal interpretability of movement scores.
[0033] Second, the limb movement evaluation method based on uncertainty-guided discriminative feature learning adopted in this invention enhances feature discrimination ability in the spatial and temporal dimensions, effectively improves the differentiation effect of fuzzy samples, and improves the model's discrimination performance on adjacent rating levels. Attached Figure Description
[0034] Figure 1This is the overall model diagram of the method proposed in the present invention.
[0035] References:
[0036] [1] Deb S., Islam M.F., Rahman S., Rahman S. “Graph Convolutional Networks for Assessment of Physical Rehabilitation Exercises.” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 30, 2022, pp. 410–419.
[0037] [2] Lei Q., Zhang H.B., Du J.X., Hsiao T.C., Chen C.C. “Learning Effective Skeletal Representations on RGB Video for Fine-Grained Human Action Quality Assessment.” Electronics, vol. 9, no. 4, 2020,
[0038] p. 568.
[0039] [3] Bian X., Chang D., Yang Y., He Z., Liang K., Ma Z. “Class-Aware Contrastive Learning for Fine-Grained Skeleton-Based Action Recognition.” In: Cho M., Laptev I., Tran D., Yao A., Zha H. (eds) Computer Vision – ACCV 2024. Lecture Notes in Computer Science, vol. 15472, Springer, Singapore, 2025, pp. 225–241.
[0040] [4]Deng D., Ostrem JL, Nguyen V., Cummins DD, Sun J., Pathak A., Little S., Abbasi-Asl R. "Interpretable Video-Based Tracking and Quantification of Parkinsonism Clinical Motor States." npj Parkinson'
[0041] s Disease,vol.10,2024,Article 122. Detailed Implementation
[0042] The following detailed description of embodiments of the present invention is provided in conjunction with the accompanying drawings:
[0043] This invention is a limb movement assessment method based on uncertainty-guided discriminative feature learning. This method introduces uncertainty-driven multi-instance aggregation and contrastive feature optimization within a spatiotemporal graph convolution framework to achieve adaptive focusing on key segments and enhanced discrimination of critical samples, thereby outputting stable, accurate and interpretable movement scoring results. The specific process of motion assessment is as follows: First, the patient's motion video is converted into a standardized skeleton sequence to establish an input structure that can be used for graph convolutional modeling. Then, the skeleton coordinates are aligned, denoised, and scaled to eliminate the influence of individual body size and shooting conditions. After normalization, an uncertain multi-instance learning branch is introduced to divide the continuous skeleton sequence into several motion instance segments of fixed or adaptive length to form the instance-level input set required for multi-instance learning. On this basis, each instance segment is modeled through a spatiotemporal graph convolutional network to extract its high-dimensional representation of spatial topology and temporal dynamics. Confidence modeling and weighted aggregation are performed on the instance-level prediction results to highlight high-confidence keyframe segments and suppress invalid or interfering segments. At the same time, a contrast feature optimization branch is constructed to decouple spatial and temporal features and complete the dynamic alignment and correction of confidence samples and fuzzy samples based on category prototypes to improve the discriminative ability between adjacent rating levels. Finally, the fusion result of the two branches is input into the scoring head for joint optimization and scoring output, and the temporal weight distribution of key segments is generated to achieve temporal interpretability and clinical traceability of the scoring process.
[0044] Specifically, the method includes the following steps:
[0045] Step 1: Video Preprocessing
[0046] The main task of video preprocessing is to transform the raw patient video data into a standardized skeleton sequence that can be used for graph convolutional modeling. Through pose estimation and coordinate alignment operations, it provides stable and uniform data input for subsequent spatiotemporal modeling. The input is the original motion evaluation video, and the output is the normalized skeleton sequence.
[0047] Step 1.1 Skeleton Sequence Extraction: The purpose of this step is to extract the spatiotemporal motion information of the human body from the video. Key points of the human skeleton are extracted frame by frame from the input video data, and a temporal skeleton sequence containing T frames and V key points is constructed according to formulas (1) and (2):
[0048]
[0049] in, Let (x, y, z) represent the 3D coordinates of the v-th joint in the t-th frame, where (x, y, z) represent the 3D spatial position of the joint in the camera coordinate system. x represents the horizontal coordinate, y represents the vertical coordinate, and z represents the depth distance of the joint from the camera's imaging plane. To ensure robustness, the system uses an OpenPose-based keypoint detection network for skeleton coordinate regression.
[0050] Step 1.2 Coordinate Normalization: Formulas (3) and (4) are used to normalize the coordinates of the skeletal sequences from different patients to eliminate the influence of differences in body shape, camera angle, and scale. The input is the original skeletal coordinates. The output is normalized coordinates.
[0051] Specifically, it includes:
[0052]
[0053] in, Let s be the centroid coordinates of all joints in this frame. t This indicates the scale of joint distribution within the frame.
[0054] Step 2: Uncertainty-Driven Multi-Instance Learning
[0055] This step introduces an uncertainty assessment mechanism to model the confidence of instance segments, adaptively identifying and focusing on key action segments, and reducing the impact of invalid or interfering segments on the global score. The input is a normalized skeleton sequence, and the output is a weighted aggregate of global features and the confidence weights of each instance.
[0056] Step 2.1 Instance Partitioning: Divide the long temporal skeleton sequence into several action instance segments {I1, I2, ..., I...} N}, where I NThis represents the Nth action instance segment, where N is the segment index used to identify the temporal order. Each instance contains a fixed or adaptive frame length L. The partitioning can employ a sliding window mechanism or a key pose detection algorithm to ensure that each instance segment fully covers a continuous motion phase. This process provides structured input for subsequent multi-instance learning.
[0057] Step 2.2 Spatiotemporal Feature Extraction: This step aims to jointly model the spatial structure and temporal dynamics of patient movements using a spatiotemporal graph convolutional network (SGMCN) to extract movement feature representations. The skeletal sequence is input into the ST-GCN backbone network. Within each frame, spatial neighborhood features are aggregated along the inherent joint topology of the human body. Convolution is performed on the joint features along the temporal dimension to capture their short-term motion patterns, resulting in a spatiotemporal feature map of the complete skeletal sequence.
[0058] Specifically, the first step is the construction of the skeleton diagram, which involves creating a space-time graph G = (V, E) for the skeletal sequence. S ∪E T ), where the set of spatial edges E S ={(v i ,v j Based on human anatomical structures (such as shoulder-elbow, hip-knee); the time edge set E is defined. T ={(v t ,v t+1 Spatial graph convolution is used to connect the correspondences of the same joint points between adjacent frames. Spatial graph convolution performs spatial convolution within a single frame to aggregate local joint features. Let the input feature matrix be... C is the number of channels and V is the number of joints. The specific formula for the output of spatial convolution is shown in (5).
[0059]
[0060] in, Represents node v i The neighborhood set of Z; i (v j ) is node v i The neighborhood normalization factor is equal to v j The number of nodes in the partition to which it belongs, used to balance the feature contributions of different partitions; f in (v j ) represents the input feature vector of the j-th joint node, which contains the spatial or high-dimensional features of that joint; This corresponds to the partition weight matrix. Temporal convolution modeling captures dynamic information between consecutive frames in the time dimension through one-dimensional convolution, where Γ represents the length of the temporal convolution kernel. Taking Γ = 9 corresponds to a local motion window of approximately 0.3 seconds. Represents joint node v i At time t and node v jThe Euclidean distance between time t+Δ is used to characterize the displacement change of the action in the time dimension; K t This represents the number of partitions in the temporal convolution, typically 3, used to learn dynamic features at different time offsets. The specific formula is defined as shown in (6).
[0061]
[0062] This operation enables the model to perceive short-term motion patterns of joints within a local time window.
[0063] Simultaneously, a learnable edge weight matrix R is introduced, and each layer of spatial graph convolution is weighted according to formula (7):
[0064] A′=(A+I)⊙R#(7)
[0065] Where A represents a fixed physical skeleton adjacency matrix, used to constrain the structural connection relationships of each joint; A′ is a learnable graph adjacency matrix, which dynamically learns non-physical associations through training; I is the identity matrix, representing the self-connection of nodes; and ⊙ is the Hadamard product. After multi-layer stacking and batch normalization, the spatiotemporal feature representation of the complete skeleton sequence is obtained. Where T′ represents the number of frames after time downsampling, used to characterize motion features at different time scales.
[0066] Step 2.3 Instance Confidence Calculation: The core purpose of this step is to distinguish the reliability of instances. Instances with high confidence correspond to key action segments, while instances with low confidence correspond to noisy segments such as pose blur, action pauses, or background interference. The input is instance features, and the output is instance score and confidence. Specifically, for each instance segment I... i Independently extract features and generate instance-level scores At the same time, the confidence level σ of this instance is calculated through uncertainty estimation. i This reflects the reliability of the segment prediction results. Uncertainty modeling is performed based on formula (8), using the Bayesian approximation approach:
[0067]
[0068] Specifically, the model parameters w include all trainable weights and biases required to construct the uncertainty-guided multi-instance learning network, consisting of spatiotemporal graph convolutional backbone network parameters, uncertainty learning layer parameters, and rating prediction layer parameters; p(y i |x i p(y) is the predicted probability distribution of the i-th instance segment; i |x i ,w) is the predicted likelihood probability of the i-th instance given the model parameters w; It is the model parameters w in the training dataset The posterior probability on; dw is the integral variable of parameter w; M is the number of samples, taken as M=10; w m It is the model parameter sample after the m-th random Dropout.
[0069] The prediction variance is estimated by multiple random Dropout forward inferences, as shown in formula (9).
[0070]
[0071] in For the average prediction result, σ i The smaller the value, the higher the confidence level.
[0072] Step 2.4 Instance Feature Fusion: Based on the confidence level of each instance, the segment features are weighted and aggregated. High-confidence segments contribute more to the global score, thus highlighting key action stages and suppressing interference from non-key segments. The input is the feature maps and corresponding confidence levels of all instances, and the output is the globally aggregated features.
[0073] Based on formula (10), the global representation is obtained by inversely weighting the confidence level of instance features:
[0074]
[0075] Where λ is the temperature coefficient, set to 1 to control the confidence sensitivity; i is the index of the action instance segment currently used for weight calculation (i = 1, 2, ..., N), and j is the summation index for all N action instance segments; σ i With σ j Each represents the prediction confidence of an action instance. This mechanism ensures that high-confidence instances have a greater weight in the global score, thereby highlighting key action phases.
[0076] Step 3: Comparative Feature Optimization
[0077] This module serves as an auxiliary structure during the training phase, aiming to enhance the model's ability to discriminate action differences between adjacent levels by leveraging contrastive learning constraints. It achieves adaptive optimization of feature layers through spatial-temporal feature decoupling, confidence sample prototype clustering, and fuzzy sample correction. The input is the spatiotemporal features of the last layer of the backbone network, and the output is the optimized feature representation (used for loss calculation).
[0078] Step 3.1 Spatial and Temporal Feature Decoupling: To enhance the discriminative power of the spatial posture structure and temporal dynamic pattern of the action, a lightweight dual-branch decoupling is performed on the spatiotemporal feature map of the last layer of the backbone network. The purpose of this operation is to separate the different dimensions of information in the spatiotemporal features, enabling the model to specifically optimize spatial topological differences and temporal rhythm differences. The input is the spatiotemporal feature map of the last layer, and the output is the decoupled spatial features and spatial features. Specifically, according to formulas (11) and (12), the spatiotemporal features F i By performing aggregation and linear transformation of the spatial and temporal channels respectively, spatial branches are obtained. With time branching features
[0079]
[0080] Pool t For time-average pooling, Pool v Spatial mean pooling is used. The two decoupled features are subjected to independent contrast constraints in their respective dimensions to highlight differences in posture and rhythm.
[0081] Step 3.2 Confidence and Fuzzy Sample Identification: The purpose of this step is to filter out reliable category prototype samples and fuzzy samples that need correction. The inputs are decoupled spatial and temporal features, instance prediction results, and the true labels of the samples. The outputs are a set of confidence samples, a set of fuzzy samples, and category prototypes. Confidence samples and fuzzy samples are dynamically identified based on the model's prediction confidence, and category-level prototype representations are established accordingly. Specifically, samples with correct predictions and high confidence are considered confidence samples, used to update the global prototype centers of each category to stabilize the feature distribution of that category. Simultaneously, samples with incorrect predictions or low confidence are considered fuzzy samples, used to assist in subsequent comparative optimization, and feature correction is achieved through distance constraints with the correct category prototypes. This mechanism can dynamically distinguish between reliable and uncertain samples during training, enabling the network to use robust category prototypes as a reference in comparative learning, thereby improving the consistency and discriminative power of feature representations.
[0082] Based on the predicted probability p i With real label y i Given an action label k, define the confidence sample set according to formula (13). Let represent the set of reliable samples where the prediction is correct and the uncertainty is below a threshold; define two types of fuzzy samples: the set of false negative samples that belong to class k but are misclassified as other classes. And the set of false positive samples that were incorrectly classified into class k.
[0083] The sample sets are the false negative and false positive samples, respectively (13).
[0084] The exponential moving average (EMA) is used to maintain the feature prototypes for each category, and the prototypes are updated smoothly using the mean of the confidence sample features to ensure their stability and representativeness. The confidence sample category prototypes are calculated according to formula (14):
[0085]
[0086] Where α is the momentum coefficient, used for smooth updates, and is taken as α = 0.9; P k F represents the global feature prototype of the k-th action class, reflecting the average representation of that class in the feature space; i This represents the high-dimensional spatiotemporal feature vector of sample i, used to update the prototype center.
[0087] Step 3.3 Prototype Comparison and Feature Correction: Feature comparison learning is performed using the class prototype as the anchor point. Confidence samples are aggregated towards their class prototype by maximizing similarity, while ambiguous samples are constrained to move away from the erroneous class prototype, thereby strengthening the inter-class distinction boundary and achieving self-correction at the feature level. The purpose of this step is to optimize the feature space distribution, making similar samples more compact and dissimilar samples more separated. The input is the feature vector F of the current confidence sample used as the comparison anchor point. i , where i is the index variable of the current confidence sample in the data batch; the category prototype P k k is the action category index; F is the feature vector F of the j-th fuzzy sample in the set of false negative and false positive samples. j , j is the index variable for traversing all elements of the set; the output is the corrected feature.
[0088] Cosine similarity is used to measure the similarity between features and prototypes. The class center distance is calculated according to formula (15):
[0089]
[0090] According to formulas (16) and (17), a compensation term φ is introduced for False Negative samples. i To make it converge towards the corresponding category prototype, the compensation term is calculated based on the feature centers of the False Negative samples.
[0091]
[0092] According to formulas (18) and (19), a penalty term ψ is introduced for False Positive samples. i To keep it away from the error category prototype, the penalty term is calculated based on the feature centers of the False Positive samples.
[0093]
[0094] By minimizing φ i ,ψ i This process gradually moves fuzzy samples closer to the center of the correct category and away from the incorrect category, thereby completing feature correction.
[0095] Step 3.4 Feature Optimization and Supervision: Integrating the corrected features from both the spatial and temporal branches, calculate the contrastive learning loss and jointly optimize it with the main task loss to enhance feature discriminativeness. The purpose of this step is to solidify the feature optimization effect through loss constraints. The input is the corrected spatial / temporal features and the class prototype, and the output is the contrastive loss L. CL Spatial features With time characteristics The comparative optimization loss CL(·) is calculated separately, and the specific formula is shown in (20).
[0096]
[0097] Where τ is a temperature parameter, set to τ = 0.1, used to adjust the scale of similarity between samples and class prototypes in contrastive learning, thereby controlling the smoothness of feature distribution. ik P represents the probability score that the model predicts the i-th sample belongs to the target class k, reflecting the confidence level of the model in classifying the current input sample as the k-th action. l Let l represent the global feature prototype of the l-th action category, where l is the index variable for traversing all action categories except the target category k (i.e., l ≠ k).
[0098] According to formula (21), the total contrastive optimization loss is defined as the sum of the spatial and temporal branch losses. The spatial and temporal decoupling feature refinement is achieved by adding the losses of the two branches:
[0099]
[0100] Where, λ l The weighting coefficients for the comparative optimization loss are represented by λ. l =1.0.
[0101] Step 4: Joint Optimization and Scoring Output
[0102] The core objective of this step is to achieve end-to-end training through multi-task joint optimization, enabling the model to maintain scoring accuracy while possessing structured interpretability. The overall optimization objective consists of two types of loss terms:
[0103] The main loss is used to constrain the model's basic scoring task to ensure the accuracy of action recognition or score prediction. The contrastive optimization loss draws on the idea of contrastive feature optimization to distinguish and correct fuzzy samples from clear samples, strengthen the consistency of intra-class features and expand the differences between classes, thereby improving the model's ability to distinguish between adjacent rating levels.
[0104] These two parts work together to enable the model to simultaneously improve feature extraction, instance aggregation, and discrimination optimization during end-to-end training, providing robust support for subsequent score prediction and interpretable output.
[0105] Step 4.1 Joint Loss Optimization: The purpose of this step is to train the entire model end-to-end through multi-task learning. The overall loss function is defined according to formula (22) and is constructed by weighted combination of the main task loss and the contrastive optimization loss. The overall optimization objective function is achieved through end-to-end training to realize joint optimization of feature extraction, instance aggregation and discrimination optimization.
[0106] L total =L cls +λ CL L CL #(twenty two)
[0107] Among them, L cls The main task loss is based on cross-entropy, and the fused feature F from step 2.4 is used. agg Calculation yielded L CL To compare the optimized loss, the hyperparameter λ is used. CL To control the weight balance, take λ. CL =0.1.
[0108] Step 4.2 Evaluation Score Prediction: The purpose of this step is to evaluate the new video using the trained model. After training, the model is given the skeleton sequence of the complete video as input and automatically outputs the final score of the patient's actions. Specifically, the global features F after fusion in step 2.4 agg The input is fed into a fully connected layer, and then through a Softmax or regression layer to obtain the final score according to formula (23):
[0109]
[0110] Where Pool(·) is a temporal pooling operation, and MLP(·) represents a multilayer perceptron structure, which aggregates the high-dimensional feature vector F. agg This is mapped to the final scoring space. Through this mechanism, the model achieves temporal interpretability while maintaining end-to-end learning. Simultaneously, the system records the confidence α for each instance segment. iThe distribution is used to form a time-series heatmap, which visualizes the weight of key segments in the time dimension, enabling the feature tracing and location of the evaluation results in the time dimension.
Claims
1. A method for assessing limb movements based on uncertainty-guided discriminative feature learning, characterized in that: Step 1: Video Preprocessing The raw human video data is transformed into a normalized skeleton sequence that can be used for graph convolutional modeling; the input is the raw motion evaluation video, and the output is the normalized skeleton sequence. Step 1.1 Skeleton Sequence Extraction: Extract the spatiotemporal motion information of the human body from the video; extract the human skeleton joint data frame by frame from the input motion evaluation video using a pose estimation algorithm to construct a complete temporal skeleton sequence; Step 1.2 Coordinate Normalization: The extracted skeleton sequence is subjected to coordinate alignment, denoising and normalization to eliminate the influence of individual body size differences and camera angle; the input is the original skeleton coordinates, and the output is the normalized coordinates; Step 2: Uncertainty-Driven Multi-Instance Learning The input is a normalized skeleton sequence, and the output is a weighted aggregate of global features and the confidence weights of each instance. Step 2.1 Instance partitioning: Divide the long temporal skeleton sequence into several short segments, i.e., instances, according to a fixed or adaptive length. The input is the complete skeleton sequence, and the output is a set of instance segments. The partitioning adopts a sliding window mechanism or a key pose detection algorithm to ensure that each instance segment completely covers a continuous motion phase. Step 2.2 Instance Feature Extraction: This step aims to jointly model the spatial structure and temporal dynamics of human movements using a spatiotemporal graph convolutional network to extract action feature representations; each instance is input into the ST-GCN backbone network, and within each frame, spatial neighborhood features are aggregated along the inherent joint topology of the human body; the joint features are convolved along the temporal dimension to capture their short-term motion patterns, resulting in a spatiotemporal feature map for each instance. Step 2.3 Instance Confidence Calculation: Features are independently extracted for each instance segment, and an instance-level score is generated. Simultaneously, the confidence score of the instance is calculated through uncertainty estimation to reflect the reliability of the segment prediction result. The input is the instance feature map, and the output is the instance's score and confidence score. Specifically, after extracting spatial and temporal features from each instance through a graph convolutional network, a compact instance representation is obtained through a global average pooling layer, and then mapped to the instance's confidence score by a linear transformation layer. This confidence score reflects the model's certainty regarding the segment prediction result. Step 2.4 Instance Feature Fusion: Based on the confidence of each instance, the segment features are weighted and aggregated using an inverse confidence weighting strategy. High-confidence segments contribute more to the global score, thus highlighting key action stages and suppressing interference from non-key segments. The input is the feature map and corresponding confidence of all instances, and the output is the global aggregated features. Step 3: Comparative Feature Optimization By embedding a contrastive feature optimization module in the last layer of the backbone network, spatial and temporal features are decoupled and prototype contrast correction is performed, enhancing the discriminativeness and robustness of the feature representation and improving the model's ability to distinguish within the critical scoring interval; the input is the spatiotemporal features of the last layer, and the output is the optimized feature representation used for loss calculation. Step 3.1 Spatial and Temporal Feature Decoupling: The purpose of this step is to separate the different dimensions of information in the spatiotemporal features, enabling the model to specifically optimize spatial topological differences and temporal rhythm differences. The input is the spatiotemporal feature map of the last layer, and the output is the decoupled spatial and temporal features. Specifically, after the multi-layer output of the backbone network, a lightweight feature optimization module is introduced to decouple the hidden spatiotemporal feature map into two branches: spatial and temporal. Specifically, the original features are fed into two parallel channels: the spatial channel performs feature aggregation in the temporal dimension to extract the topological structure and body posture distribution information between joints; the temporal channel performs feature aggregation in the spatial dimension to extract the dynamic changes and rhythmic patterns between frames during the action. Subsequently, each branch performs linear transformation and normalization on the extracted features to obtain the spatial and temporal features. Step 3.2 Confidence and Fuzzy Sample Identification: The purpose of this step is to screen reliable category prototype samples and fuzzy samples that need correction. The inputs are decoupled spatial and temporal features, instance prediction results, and the true labels of the samples. The outputs are a set of confidence samples, a set of fuzzy samples, and category prototypes. Confidence samples and fuzzy samples are dynamically identified based on the model's prediction confidence, and category-level prototype representations are established accordingly. Specifically, samples that are correctly predicted and have high confidence are considered as confidence samples and are used to update the global prototype centers of each category to stably represent the feature distribution of that category. At the same time, samples that are incorrectly predicted or have low confidence are considered as fuzzy samples and are used to assist in subsequent comparative optimization. Feature correction is achieved through distance constraints with the correct category prototypes. Step 3.3 Prototype Comparison and Feature Correction: Feature comparison learning is performed using category prototypes as anchors; confidence samples are aggregated to their respective category prototypes by maximizing similarity, and fuzzy samples are constrained to move away from erroneous category prototypes, thereby strengthening the inter-class distinction boundary and achieving self-correction at the feature level; the inputs are decoupled features, category prototypes, and fuzzy samples, and the output is the corrected features; Step 3.4 Feature Optimization and Supervision: Integrate the corrected features from the spatial and temporal branches, calculate the contrastive learning loss, and jointly optimize it with the main task loss to enhance feature discriminativeness. The purpose of this step is to solidify the feature optimization effect through loss constraints. The inputs are the corrected spatial features, temporal features, and class prototypes, and the output is the contrastive loss. Step 4: Joint Optimization and Scoring Output This step aims to integrate multi-task losses, achieve end-to-end optimized training, and output the final action score results and interpretability information; Step 4.1 Joint Loss Optimization: The purpose of this step is to train the entire model end-to-end through multi-task learning. During the model training phase, end-to-end learning is achieved through a multi-task joint optimization strategy. The overall optimization objective consists of two types of loss terms: The main loss is used to constrain the model's basic scoring task to ensure the accuracy of action recognition or score prediction; the contrastive optimization loss draws on the idea of contrastive feature optimization to distinguish and correct fuzzy samples from clear samples, strengthen the consistency of intra-class features and expand the differences between classes, thereby improving the model's ability to distinguish between adjacent rating levels. Step 4.2 Evaluation Score Prediction: The purpose of this step is to evaluate the new video using the trained model. After the model is trained, the skeletal sequence of the complete video is input, the final score of the action is output, and the time position and weight distribution of the key segments are given.
2. The method according to claim 1, characterized in that: Step 1: Video Preprocessing The raw human video data is transformed into a standardized skeleton sequence that can be used for graph convolutional modeling. Through pose estimation and coordinate alignment operations, it provides stable and uniform data input for subsequent spatiotemporal modeling. The input is the original motion evaluation video, and the output is the normalized skeleton sequence. Step 1.1 Skeleton Sequence Extraction: Extract human skeleton key points frame by frame from the input video data, and construct a temporal skeleton sequence containing T frames and V key points according to formulas (1) and (2): in, Let (x, y, z) represent the three-dimensional coordinates of the v-th joint in the t-th frame. (x, y, z) represent the three-dimensional spatial position information of the joint in the camera coordinate system. x represents the horizontal coordinate position, y represents the vertical coordinate position, and z represents the depth distance of the joint from the camera imaging plane. To ensure robustness, the system uses a keypoint detection network based on OpenPose for skeleton coordinate regression. Step 1.2 Coordinate Normalization: Use formulas (3) and (4) to standardize the coordinates of different human skeleton sequences to eliminate the influence of differences in body shape, camera angle, and scale; the input is the original skeleton coordinates. The output is normalized coordinates. Specifically, it includes: in, Let s be the centroid coordinates of all joints in this frame. t Indicates the scale of intra-frame joint distribution; Step 2: Uncertainty-Driven Multi-Instance Learning This step introduces an uncertainty assessment mechanism to model the confidence of instance segments, adaptively identifying and focusing on key action segments, and reducing the impact of invalid or interfering segments on the global score; the input is a normalized skeleton sequence, and the output is a weighted aggregated global feature and the confidence weight of each instance; Step 2.1 Instance Partitioning: Divide the long temporal skeleton sequence into several action instance segments {I1, I2, ..., I...} N }, where I N This represents the Nth action instance segment, where N is the segment index used to identify the temporal order; each instance contains a fixed or adaptive frame length L; the partitioning can use a sliding window mechanism or a key pose detection algorithm to ensure that each instance segment completely covers a continuous motion phase; Step 2.2 Spatiotemporal Feature Extraction: This step aims to jointly model the spatial structure and temporal dynamics of human movements using a spatiotemporal graph convolutional network to extract movement feature representations; input the skeletal sequence into the ST-GCN backbone network, and within each frame, aggregate spatial neighborhood features along the inherent joint topology of the human body; convolve the joint features along the temporal dimension to capture their short-term motion patterns, and obtain the spatiotemporal feature map of the complete skeletal sequence; The first step is to construct the skeleton diagram, creating a space-time graph G = (V, E) for the skeletal sequence. S ∪E T ), where the set of spatial edges E S ={(v i ,v j Based on human anatomical structures (such as shoulder-elbow, hip-knee); the time edge set E is defined. T ={(v t ,v t+1 Spatial graph convolution is used to connect the correspondences of the same joint points between adjacent frames; spatial graph convolution is performed within a single frame to aggregate local joint features; let the input feature matrix be... C is the number of channels, V is the number of joints, and the specific formula for the output of spatial convolution is shown in (5); in, Represents node v i The neighborhood set of Z; i (v j ) is node v i The neighborhood normalization factor is equal to v j The number of nodes in the partition to which it belongs, used to balance the feature contributions of different partitions; f in (v j ) represents the input feature vector of the j-th joint node, which contains the spatial or high-dimensional features of that joint; This corresponds to the partition weight matrix; temporal convolution modeling captures dynamic information between consecutive frames in the time dimension through one-dimensional convolution, where Γ represents the length of the temporal convolution kernel, and Γ = 9 corresponds to a local motion window of about 0.3 seconds; Represents joint node v i At time t and node v j The Euclidean distance between time t+Δ is used to characterize the displacement change of the action in the time dimension; K t The value of 3 represents the number of partitions in the temporal convolution, used to learn dynamic features at different time offsets; the specific formula is defined as shown in (6); This operation enables the model to perceive short-term motion patterns of joints within a local time window; Simultaneously, a learnable edge weight matrix R is introduced, and each layer of spatial graph convolution is weighted according to formula (7): A′=(A+I)⊙R# (7) Where A represents a fixed physical skeleton adjacency matrix, used to constrain the structural connection relationships of each joint; A′ is a learnable graph adjacency matrix, which dynamically learns non-physical associations through training; I is the identity matrix, representing the self-connection of nodes; ⊙ is the Hadamard product; after multi-layer stacking and batch normalization, the spatiotemporal feature representation of the complete skeleton sequence is obtained. Where T′ represents the number of frames after time downsampling, used to characterize motion features at different time scales; Step 2.3 Instance Confidence Calculation: The core purpose of this step is to distinguish the reliability of instances. Instances with high confidence correspond to key action segments, while instances with low confidence correspond to noisy segments such as pose blur, action pauses, or background interference. The input is instance features, and the output is instance score and confidence. Specifically, for each instance segment I... i Independently extract features and generate instance-level scores At the same time, the confidence level σ of this instance is calculated through uncertainty estimation. i To reflect the credibility of the segment prediction results; uncertainty modeling is performed according to formula (8), using the Bayesian approximation idea: Specifically, the model parameters w include all trainable weights and biases required to construct the uncertainty-guided multi-instance learning network, consisting of spatiotemporal graph convolutional backbone network parameters, uncertainty learning layer parameters, and rating prediction layer parameters; p(y i |x i p(y) is the predicted probability distribution of the i-th instance segment; i |x i ,w) is the predicted likelihood probability of the i-th instance given the model parameters w; It is the model parameters w in the training dataset The posterior probability on; dw is the integral variable of parameter w; M is the number of samples, taken as M=10; w m These are the model parameter samples after the m-th random Dropout; The prediction variance is estimated by multiple random Dropout forward inferences, and the specific formula is shown in (9); in For the average prediction result, σ i The smaller the value, the higher the confidence level. Step 2.4 Instance Feature Fusion: Based on the confidence of each instance, the segment features are weighted and aggregated. High-confidence segments contribute more to the global score, thus highlighting key action stages and suppressing the interference of non-key segments; the input is the feature map and corresponding confidence of all instances, and the output is the global aggregated features; Based on formula (10), the global representation is obtained by inversely weighting the confidence level of instance features: Where λ is the temperature coefficient, set to 1 to control the confidence sensitivity; i is the index of the action instance segment currently used for weight calculation (i = 1, 2, ..., N), and j is the summation index for all N action instance segments; σ i With σ j Both represent the prediction confidence of an action instance; Step 3: Comparative Feature Optimization Adaptive optimization of the feature level is achieved through spatial-temporal feature decoupling, confidence sample prototype clustering, and fuzzy sample correction; the input is the spatiotemporal features of the last layer of the backbone network, and the output is the optimized feature representation used for loss calculation. Step 3.1 Decoupling Spatial and Temporal Features: To enhance the discriminative power of the spatial posture structure and temporal dynamic pattern of the action, a lightweight dual-branch decoupling is performed on the spatiotemporal feature map of the last layer of the backbone network. The purpose of this operation is to separate the different dimensions of information in the spatiotemporal features, so that the model can specifically optimize the differences in spatial topology and temporal rhythm. The input is the spatiotemporal feature map of the last layer, and the output is the decoupled spatial features and spatial features. Specifically, according to formulas (11) and (12), the spatiotemporal features F are decoupled. i By performing aggregation and linear transformation of the spatial and temporal channels respectively, spatial branches are obtained. With time branching features Pool t For time-average pooling, Pool v Spatial mean pooling; Step 3.2 Confidence and Fuzzy Sample Identification: The purpose of this step is to screen reliable category prototype samples and fuzzy samples that need correction. The inputs are decoupled spatial and temporal features, instance prediction results, and the true labels of the samples. The outputs are a set of confidence samples, a set of fuzzy samples, and category prototypes. Based on the model prediction confidence, confidence samples and fuzzy samples are dynamically identified, and category-level prototype representations are established accordingly. Specifically, samples that are correctly predicted and have high confidence are considered as confidence samples and are used to update the global prototype centers of each category to stably represent the feature distribution of that category. At the same time, samples that are incorrectly predicted or have low confidence are considered as fuzzy samples and are used to assist in subsequent comparison and optimization. Feature correction is achieved through distance constraints with the correct category prototypes. Based on the predicted probability p i With real label y i Given an action label k, define the confidence sample set according to formula (13). Let represent the set of reliable samples where the prediction is correct and the uncertainty is below a threshold; define two types of fuzzy samples: the set of false negative samples that belong to class k but are misclassified as other classes. And the set of false positive samples that were incorrectly classified into class k. The exponential moving average (EMA) is used to maintain the feature prototypes of each category, and the prototypes are updated smoothly with the mean of the confidence sample features to ensure the stability and representativeness of the prototypes; the confidence sample category prototypes are calculated according to formula (14): Where α is the momentum coefficient, used for smooth updates, and is taken as α = 0.9; P k F represents the global feature prototype of the k-th action class, reflecting the average representation of that class in the feature space; i This represents the high-dimensional spatiotemporal feature vector of sample i, used to update the prototype center; Step 3.3 Prototype Comparison and Feature Correction: Feature comparison learning is performed using the category prototype as the anchor point; confidence samples are aggregated towards their respective category prototypes by maximizing similarity, while ambiguous samples are constrained to move away from the erroneous category prototypes, thereby strengthening the inter-class distinction boundary and achieving self-correction at the feature level; the purpose of this step is to optimize the feature space distribution, making similar samples more compact and dissimilar samples more separated. The input is the feature vector F of the current confidence sample used as the comparison anchor point. i , where i is the index variable of the current confidence sample in the data batch; the category prototype P k , k is the action category index; F is the feature vector F of the j-th fuzzy sample in the set of false negative and false positive samples. j j is the index variable for traversing all elements of the set; the output is the corrected feature. Cosine similarity is used to measure the similarity between features and prototypes. The class center distance is calculated according to formula (15): According to formulas (16) and (17), a compensation term φ is introduced for False Negative samples. i To make it converge towards the corresponding category prototype, the compensation term is calculated based on the feature centers of the False Negative samples. According to formulas (18) and (19), a penalty term ψ is introduced for False Positive samples. i To keep it away from the error category prototype, the penalty term is calculated based on the feature centers of the False Positive samples. By minimizing φ i ,ψ i This process gradually moves fuzzy samples closer to the center of the correct category and away from the incorrect category, thereby completing feature correction. Step 3.4 Feature Optimization and Supervision: Integrating the corrected features from both the spatial and temporal branches, calculate the contrastive learning loss and jointly optimize it with the main task loss to enhance feature discriminativeness. The purpose of this step is to solidify the feature optimization effect through loss constraints. The input is the corrected spatial / temporal features and the class prototype, and the output is the contrastive loss L. CL Spatial features With time characteristics The comparative optimization loss CL(·) is calculated separately, and the specific formula is shown in (20); Where τ is a temperature parameter, taken as τ = 0.1, used to adjust the scale of similarity between samples and class prototypes in contrastive learning, thereby controlling the smoothness of feature distribution; p ik P represents the probability score that the model predicts the i-th sample belongs to the target class k, reflecting the confidence level of the model in classifying the current input sample as the k-th action; l Let l represent the global feature prototype of the l-th action category, where l is the index variable for traversing all action categories except the target category k, i.e., l ≠ k; According to formula (21), the total contrastive optimization loss is defined as the sum of the spatial and temporal branch losses. The spatial and temporal decoupling feature refinement is achieved by adding the losses of the two branches: Where, λ l The weighting coefficients for the comparative optimization loss are represented by λ. l =1.0; Step 4: Joint Optimization and Scoring Output The core objective of this step is to achieve end-to-end training through multi-task joint optimization, enabling the model to maintain scoring accuracy while possessing structured interpretability; the overall optimization objective consists of two types of loss terms: The main loss is used to constrain the model's basic scoring task to ensure the accuracy of action recognition or scoring prediction; the contrastive optimization loss draws on the idea of contrastive feature optimization to distinguish and correct fuzzy samples from clear samples, strengthen the consistency of intra-class features and expand the differences between classes. Step 4.1 Joint Loss Optimization: The purpose of this step is to train the entire model end-to-end through multi-task learning. The overall loss function is defined by formula (22) and is constructed by weighted combination of the main task loss and the contrast optimization loss. The joint optimization of feature extraction, instance aggregation and discrimination optimization is achieved through end-to-end training. L total =L cls +λ CL L CL #(22) Among them, L cls The main task loss is based on cross-entropy, and the fused feature F from step 2.4 is used. agg Calculation yielded L CL To compare the optimized loss; hyperparameter λ CL To control the weight balance, take λ. CL =0.1; Step 4.2 Evaluation Score Prediction: The purpose of this step is to evaluate the new video using the trained model. After training, the model is given the skeleton sequence of the complete video as input and automatically outputs the final score for the human body movements. Specifically, the global features F after fusion in step 2.4 agg The input is fed into a fully connected layer, and then through a Softmax or regression layer to obtain the final score according to formula (23): Where Pool(·) is a temporal pooling operation, and MLP(·) represents a multilayer perceptron structure, which aggregates the high-dimensional feature vector F. agg Mapped to the final score space; the confidence α of each instance segment is recorded. i Distribution, forming a time-series heatmap.
