Multi-stage motion analysis method for motion quality evaluation
Through the multi-level motion analysis method, joint-level motion characteristics were extracted and combined with custom weight scores, the problem of difficult joint-level motion patterns in motion quality evaluation was solved, and higher scoring accuracy and stability were achieved.
Patent Information
- Application Number
- CN202510540205.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-27
- Publication Date
- 2025-08-08
AI Technical Summary
The prior art is difficult to accurately capture joint-level motion patterns and subtle changes in motion quality evaluation, resulting in insufficient accuracy of motion segmentation and scoring, especially in complex backgrounds.
The multi-level motion analysis method is adopted to extract the skeleton information and the relative position changes of joints through the action unit parser, combine the fusion of color and skeleton characteristics, and use a custom weight scoring module to perform score differences regression, and introduce a perfect comparison mechanism and a multi-instance balanced voting mechanism to improve scoring accuracy.
It improves the accuracy and interpretability of action quality assessment, complies with international scoring standards, and enhances the generalization ability of the method and the stability of the predicted results.
Smart Images
Figure CN120449121A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the interdisciplinary field of computer vision and artificial intelligence, and specifically relates to a multi-level motion analysis method for motion quality assessment, which is suitable for scenarios such as sports events and rehabilitation training that require high-precision motion analysis and scoring. Background Art
[0002] With the continuous development of computer vision technology, video understanding has become an effective auxiliary tool for action quality assessment (AQA) in the sports field. In particular, in competitions such as diving, gymnastics, and figure skating, intelligent AQA has many advantages, such as avoiding subjective scoring errors and providing quantitative guidance on action details. However, sports videos often contain various backgrounds, high-speed switching, and different athletes. To date, direct action assessment has remained a challenging problem. These difficulties have prompted us to rethink how to improve the accuracy of AQA and action segmentation tasks by starting from the action itself.
[0003] In the field of AQA, early studies relied on manual feature extraction, such as regression based on human pose encoding or support vector regression (SVR), but it was difficult to capture temporal dynamic information. With the development of deep learning, spatiotemporal feature modeling methods such as C3D, I3D, and LSTM have enhanced the ability to model motion trajectories, pushing the AQA task towards direct regression and pairwise comparison methods. Yu et al. [3] first proposed a pairwise comparison model to learn subtle differences between actions and improve the performance and interpretability of the AQA task. An et al. further proposed a multi-stage contrastive regression framework to more effectively extract spatiotemporal features. Although these methods have made some progress in feature extraction and scoring consistency, most of them rely on global video features and do not deeply model joint-level motion patterns. In addition, in actual competitions, important factors affecting scoring also include splash size and gymnastics landing stability.
[0004] In the field of action segmentation, traditional methods rely on time series modeling, such as long short-term memory networks (LSTMs) and transformers, to predict motion boundaries by learning inter-frame correlations. However, these methods focus on global features and lack segmentation accuracy in scenes with complex backgrounds or subtle motion changes. Yu et al. proposed a new action segmentation framework, ASRF, to reduce over-segmentation errors by detecting motion boundaries. Zhang et al. proposed a multi-hidden sub-stage learning and fusion network, introducing a semantic segmentation model for stage division. Li et al. proposed a multi-stage temporal convolutional network (MS-TCN) architecture that reduces the number of parameters while ensuring a large receptive field. To improve accuracy, researchers introduced a regional attention mechanism to optimize the segmentation effect of key regions. Liu et al. combined bidirectional temporal input with a multi-stage interactive segmentation-aware graph convolutional network (GCN) to further improve sparse action segmentation performance. However, these methods have not yet focused on the key elements of the action, especially the relative positions of joints and subtle changes in body bending angles. Summary of the Invention
[0005] To address the above-mentioned problems in the prior art, the present invention proposes a multi-level motion analysis method for action quality assessment. By focusing on the action itself (i.e., changes in joint relative positions and joint angles) to extract more detailed and effective spatiotemporal motion features, the accuracy of AQA and action segmentation tasks can be effectively improved.
[0006] In order to achieve the above object, the present invention adopts the following technical solutions:
[0007] A multi-stage motion parsing method for motion quality assessment, including training and prediction processes;
[0008] The training process runs as follows:
[0009] Step 1: Input a video whose score needs to be queried. A perfect comparison mechanism is used to select a reference video corresponding to the query video and form a video pair with the query video, and then input it into the action unit parser.
[0010] Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them;
[0011] Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively;
[0012] Step 4: Use the custom weight scoring module to combine different features at different stages to perform regression calculations on the score differences, and use this to obtain the predicted score of the query video;
[0013] Step 5: Construct the loss function and use the Nadam optimizer with a learning rate of 0.0001 and automatic backpropagation training with the help of cosine annealing training method;
[0014] The prediction process runs as follows:
[0015] Step 1: Input a video whose score needs to be queried. Use a multi-instance balanced voting mechanism to select a corresponding number of reference videos to form multiple video pairs with the query video. Input the multiple video pairs into the action unit parser in sequence.
[0016] Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them;
[0017] Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively;
[0018] Step 4: Use the custom weight scoring module to perform regression calculations on the score differences based on the different features at different stages, and use this to obtain the predicted score of the query video in the video pair;
[0019] Step 5: Average the multiple prediction scores obtained from multiple video pairs input, and the result is the final score of the predicted video.
[0020] The operation steps of the action unit parser include:
[0021] Step 1: Use the pose extractor of the YOLOv11 architecture to extract the skeleton coordinate information and bounding box coordinate information of the main athlete in the corresponding video;
[0022] Step 2: Fuse the bounding box coordinates. If the number of bounding boxes in each frame exceeds 1, all bounding boxes are fused. The upper left corner of the fused bounding box is the minimum value of the upper left corner coordinates of all bounding boxes, and the lower right corner coordinates is the maximum value of the lower right corner coordinates of all bounding boxes, so that it can cover the main athlete area as much as possible.
[0023] Step 3: Generate action unit representation images for each frame of the video based on the skeleton coordinates and image resolution. Specifically, for each frame, select the shoulders, hips, knees, and ankles from the skeleton coordinates, one for each part, for a total of four pairs of coordinates. If a required coordinate does not exist, it is represented as a null value. The steps are as follows:
[0024] Step 3.1: First, calculate the size of the action unit image. All action unit images are square, and their width and height are expressed as CutSize. The calculation method of CutSize is as follows:
[0025]
[0026] Among them, width box and height box are the width and height of the bounding box of the player area in the first frame of the sequence, width image and height image are the overall width and height of the frame image respectively;
[0027] Step 3.2: Generate a black canvas with a width of 4 × CutSize pixels and a height of 2 × CutSize pixels. Divide it into two parts according to its width. In the left part, generate the region of interest images of the shoulder, hip, knee, and ankle from top to bottom and from left to right. Each region of interest is generated by intercepting a region of size CutSize × CutSize pixels centered on the average center coordinate of a pair of coordinates of the part. In the right part, intercept the entire bounding box area and resize the image to a rectangle with a width of 2 × CutSize pixels and a height of 2 × CutSize pixels.
[0028] Step 3.3: Change the overall size of the stitched image to 128×128 pixels;
[0029] Step 4: Use the bounding box coordinate information to perform rectangular cropping on the video image;
[0030] Step 5: Input the cropped image sequence into a color feature analysis module 1 composed of an I3D network to obtain color features;
[0031] Step 6: Input the skeleton information sequence corresponding to the image sequence described in step 5 into a skeleton sequence parsing module 1 composed of LSTM, Transformer-Encoder and Nystrom-Attention to obtain skeleton feature 1. The operation steps of the skeleton sequence parsing module are as follows:
[0032] Step 6.1: Input the skeleton information sequence into Nystrom-Attention, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain the first-level skeleton feature X1;
[0033] Step 6.2: Input the skeleton information sequence into LSTM, then input the output of LSTM into Nystrom-Attention, and finally input the output of Nystrom-Attention into Transformer-Encoder to obtain the secondary skeleton feature X2;
[0034] Step 6.3: Concatenate the first-level skeleton feature X1 and the second-level skeleton feature X2 with the dimension 1 of the feature tensor to obtain the final skeleton feature 1;
[0035] Step 7: Stack the color features and the final skeleton features in the time dimension and regress them into a probability matrix through a 3-layer simple MLP. The number of rows R of this probability matrix represents the division of a motion into R+1 stages, the number of columns C represents the total number of frames after video alignment, and the number in the rth row and cth column represents the probability of transitioning from the rth stage to the r+1th stage occurring in the cth frame.
[0036] Step 8: Divide the skeleton information set into R+1 stages according to the probability matrix, divide the motion unit set into R+2 stages, which includes a conditional stage, which is the splash stage in diving, and align each stage.
[0037] The structure and operation steps of the motion analyzer are as follows:
[0038] The structure contains:
[0039] Color feature analysis module 2: consists of I3D network;
[0040] Skeleton feature parsing module 2: It consists of three components: Transformer-Encoder, LSTM and Nystrom-Attention.
[0041] Run steps:
[0042] Step 1: Input the action units of the first R+1 stages, i.e., the set of action units excluding the conditional stage, into the second color feature analysis module to obtain color features;
[0043] Step 2: Input the skeleton information set of the first R+1 stages, that is, excluding the conditional stage, into the skeleton feature parser 2 to obtain skeleton feature 2; the operation process in the skeleton feature parsing module 2 is as follows:
[0044] Step 2.1: Input the skeleton information set into Nystrom-Attention, and then input the output of Nystrom-Attention into LSTM to obtain the primary skeleton features;
[0045] Step 2.2: Input the primary skeleton features into Nystrom-Attention again, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain skeleton features;
[0046] Step 3: Stack and fuse the color feature 2 and the skeleton feature 2 in the time dimension to finally obtain the motion feature.
[0047] The structure and operation steps of the conditional parser are as follows:
[0048] The structure includes: I3D network as the skeleton network.
[0049] Running steps: Input the R+2th stage, that is, the set of action units in the conditional stage, to obtain the conditional features.
[0050] The structure and operation steps of the custom weight scoring module are as follows:
[0051] The structure contains:
[0052] Feature Difference Parsing Module: This is a ViT-Decoder that applies a cross-attention mechanism;
[0053] Score regression module: contains a 3-layer simple MLP network;
[0054] Custom weight score fusion module;
[0055] Run steps:
[0056] Step 1: Input the motion features obtained from the query video and the reference video into the feature difference analysis module to obtain the motion feature differences corresponding to the two videos;
[0057] Step 2: Input the conditional features obtained from the query video and the reference video into the feature difference analysis module to obtain the conditional feature differences corresponding to the two videos;
[0058] Step 3: Input the motion feature difference and condition feature difference into the score regression module to obtain the motion part score difference and condition part score difference respectively;
[0059] Step 4: Add the score difference of the sports part and the score difference of the condition part according to the custom weight to obtain the total score difference;
[0060] Step 5: Add the total score difference to the true score of the reference video to obtain the final prediction score of the query video.
[0061] The description of the perfect comparison mechanism is as follows:
[0062] A multi-process variable ReferenceDict is introduced during the training process to maximize the generalization performance when performing action quality assessment; this variable is used to store a list of video names that have been compared with each query video for training; in each training cycle, videos that are not in the list, that is, videos that have not been compared with the query video, are selected for comparison training.
[0063] The multi-instance balanced voting mechanism is described as follows:
[0064] For each prediction, 10 reference videos with the same action type as the query video are selected from the list of the user-defined training set for score prediction. The prediction scores obtained based on these 10 reference videos are averaged and the calculated average value is used as the final prediction score. The reference video selection strategy is as follows: first, all videos matching the user-specified query video are filtered out from the user-defined training set based on the action type; then, all filtered videos are sorted according to the absolute value of the original frame number difference between each filtered video and the user-specified query video; finally, the first 10 videos from the sorted video sequence are selected as reference videos.
[0065] The loss function is constructed as follows:
[0066] The total loss function is as follows:
[0067]
[0068] Used for optimized action segmentation in the action unit parser, the calculation method is as follows:
[0069] BCE(x,y)=-(x log y+(1-x)log(1-y))
[0070]
[0071] in, is the true probability of the hth action transition occurring in the tth frame; relatively, is the predicted probability of the zth action transition occurring in the tth frame; and The formula for the entire score evaluation framework is as follows:
[0072]
[0073] in, is the prediction score output by the model, X Q is the true score of the query video.
[0074] Compared with the prior art, the present invention has the following advantages:
[0075] 1. Use the action unit parser to analyze the athlete's skeleton information in the video, and use the skeleton feature parsing module to analyze subtle skeleton features such as changes in the relative position of joint points and changes in joint bending angles. The application of this feature allows the parser to focus on the key parts of the athlete's body and greatly enhances the interpretability and reliability of the method.
[0076] 2. Using the fusion of skeleton features and color features to perform motion stage division, it outperforms previous mainstream methods in terms of stage division accuracy.
[0077] 3. Use motion parser and condition parser to separate motion features and condition features (such as splashes in diving) in sports, and fuse them according to specific weights to calculate the final score, which meets international scoring standards.
[0078] 4. Use the "perfect comparison mechanism" to promote the training process and maximize the generalization ability of the method in score prediction.
[0079] 5. Use the "multi-instance balanced voting mechanism" to improve the accuracy of prediction results and reduce the instability of the results. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] Figure 1 Flow chart of the method of the present invention.
[0081] Figure 2 Schematic diagram of the selection method of action unit image elements.
[0082] Figure 3 This is a diagram of the action segmentation results of the method of the present invention on the FineDiving dataset, a dataset publicly available in the art. DETAILED DESCRIPTION
[0083] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0084] like Figure 1 As shown, the present invention provides a multi-level motion analysis method for motion quality assessment, including a training process and a prediction process; the operating steps of the training process are as follows:
[0085] Step 1: Input a video whose score needs to be queried. A perfect comparison mechanism is used to select a reference video corresponding to the query video and form a video pair with the query video, and then input it into the action unit parser.
[0086] Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them;
[0087] Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively;
[0088] Step 4: Use the custom weight scoring module to combine different features at different stages to perform regression calculations on the score differences, and use this to obtain the predicted score of the query video;
[0089] Step 5: Construct the loss function and use the Nadam optimizer with a learning rate of 0.0001 and automatic backpropagation training with the help of cosine annealing training method;
[0090] The prediction process runs as follows:
[0091] Step 1: Input a video whose score needs to be queried. Use a multi-instance balanced voting mechanism to select a corresponding number of reference videos to form multiple video pairs with the query video. Input the multiple video pairs into the action unit parser in sequence.
[0092] Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them;
[0093] Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively;
[0094] Step 4: Use the custom weight scoring module to perform regression calculations on the score differences based on the different features at different stages, and use this to obtain the predicted score of the query video in the video pair;
[0095] Step 5: Average the multiple prediction scores obtained from multiple video pairs input, and the result is the final score of the predicted video.
[0096] The operation steps of the action unit parser include:
[0097] Step 1: Use the pose extractor of the YOLOv11 architecture to extract the skeleton coordinate information and bounding box coordinate information of the main athlete in the corresponding video;
[0098] Step 2: Fuse the bounding box coordinates. If the number of bounding boxes in each frame exceeds 1, all bounding boxes are fused. The upper left corner of the fused bounding box is the minimum value of the upper left corner coordinates of all bounding boxes, and the lower right corner coordinates is the maximum value of the lower right corner coordinates of all bounding boxes, so that it can cover the main athlete area as much as possible.
[0099] Step 3: Generate the action unit representation image corresponding to each frame of the video based on the skeleton coordinates and image resolution. The selection method is as follows: Figure 2 Specifically, for each frame, select the shoulders, hips, knees, and ankles in the skeleton coordinates, one for each of the left and right parts, for a total of four pairs of coordinates. If a required coordinate does not exist, it is represented as a null value. The steps are as follows:
[0100] Step 3.1: First, calculate the size of the action unit image. All action unit images are square, and their width and height are expressed as CutSize. The calculation method of CutSize is as follows:
[0101]
[0102] Among them, width box and height box are the width and height of the bounding box of the player area in the first frame of the sequence, width image and height image are the overall width and height of the frame image respectively;
[0103] Step 3.2: Generate a black canvas with a width of 4 × CutSize pixels and a height of 2 × CutSize pixels. Divide it into two parts according to its width. In the left part, generate the region of interest images of the shoulder, hip, knee, and ankle from top to bottom and from left to right. Each region of interest is generated by intercepting a region of size CutSize × CutSize pixels centered on the average center coordinate of a pair of coordinates of the part. In the right part, intercept the entire bounding box area and resize the image to a rectangle with a width of 2 × CutSize pixels and a height of 2 × CutSize pixels.
[0104] Step 3.3: Change the overall size of the stitched image to 128×128 pixels;
[0105] Step 4: Use the bounding box coordinate information to perform rectangular cropping on the video image;
[0106] Step 5: Input the cropped image sequence into a color feature analysis module 1 composed of an I3D network to obtain color features;
[0107] Step 6: Input the skeleton information sequence corresponding to the image sequence described in step 5 into a skeleton sequence parsing module 1 composed of LSTM, Transformer-Encoder and Nystrom-Attention to obtain skeleton feature 1. The operation steps of the skeleton sequence parsing module are as follows:
[0108] Step 6.1: Input the skeleton information sequence into Nystrom-Attention, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain the first-level skeleton feature X1.
[0109] Step 6.2: Input the skeleton information sequence into LSTM, then input the output of LSTM into Nystrom-Attention, and finally input the output of Nystrom-Attention into Transformer-Encoder to obtain the secondary skeleton feature X2.
[0110] Step 6.3: Concatenate the first-level skeleton feature X1 and the second-level skeleton feature X2 with the dimension 1 of the feature tensor to obtain the final skeleton feature one.
[0111] Step 7: Stack the color features and the final skeleton features in the time dimension and regress them into a probability matrix through a 3-layer simple MLP. The number of rows R of this probability matrix represents the division of a motion into R+1 stages, the number of columns C represents the total number of frames after video alignment, and the number in the rth row and cth column represents the probability of transitioning from the rth stage to the r+1th stage occurring in the cth frame.
[0112] Step 8: Divide the skeleton information set into R+1 stages according to the probability matrix, divide the motion unit set into R+2 stages, which includes a conditional stage, which is the splash stage in diving, and align each stage.
[0113] The structure and operation steps of the motion analyzer are as follows:
[0114] The structure contains:
[0115] Color feature analysis module 2: consists of I3D network;
[0116] Skeleton feature parsing module 2: It consists of three components: Transformer-Encoder, LSTM and Nystrom-Attention.
[0117] Run steps:
[0118] Step 1: Input the action units of the first R+1 stages, i.e., the set of action units excluding the conditional stage, into the second color feature analysis module to obtain color features;
[0119] Step 2: Input the skeleton information set of the first R+1 stages, that is, excluding the conditional stage, into the skeleton feature parser 2 to obtain skeleton feature 2; the operation process in the skeleton feature parsing module 2 is as follows:
[0120] Step 2.1: Input the skeleton information set into Nystrom-Attention, and then input the output of Nystrom-Attention into LSTM to obtain the primary skeleton features;
[0121] Step 2.2: Input the primary skeleton features into Nystrom-Attention again, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain skeleton features;
[0122] Step 3: Stack and fuse the color feature 2 and the skeleton feature 2 in the time dimension to finally obtain the motion feature.
[0123] The structure and operation steps of the conditional parser are as follows:
[0124] The structure includes: I3D network as the skeleton network.
[0125] Running steps: Input the R+2th stage, that is, the set of action units in the conditional stage, to obtain the conditional features.
[0126] The structure and operation steps of the custom weight scoring module are as follows:
[0127] The structure contains:
[0128] Feature Difference Parsing Module: This is a ViT-Decoder that applies a cross-attention mechanism;
[0129] Score regression module: contains a 3-layer simple MLP network;
[0130] Custom weight score fusion module;
[0131] Run steps:
[0132] Step 1: Input the motion features obtained from the query video and the reference video into the feature difference analysis module to obtain the motion feature differences corresponding to the two videos;
[0133] Step 2: Input the conditional features obtained from the query video and the reference video into the feature difference analysis module to obtain the conditional feature differences corresponding to the two videos;
[0134] Step 3: Input the motion feature difference and condition feature difference into the score regression module to obtain the motion part score difference and condition part score difference respectively;
[0135] Step 4: Add the score difference of the sports part and the score difference of the condition part according to the custom weight to obtain the total score difference;
[0136] Step 5: Add the total score difference to the true score of the reference video to obtain the final prediction score of the query video.
[0137] The description of the perfect comparison mechanism is as follows:
[0138] A multi-process variable ReferenceDict is introduced during the training process to maximize the generalization performance when performing action quality assessment; this variable is used to store a list of video names that have been compared with each query video for training; in each training cycle, videos that are not in the list, that is, videos that have not been compared with the query video, are selected for comparison training.
[0139] The multi-instance balanced voting mechanism is described as follows:
[0140] For each prediction, 10 reference videos with the same action type as the query video are selected from the list of the user-defined training set for score prediction. The prediction scores obtained based on these 10 reference videos are averaged and the calculated average value is used as the final prediction score. The reference video selection strategy is as follows: first, all videos matching the user-specified query video are filtered out from the user-defined training set based on the action type; then, all filtered videos are sorted according to the absolute value of the original frame number difference between each filtered video and the user-specified query video; finally, the first 10 videos from the sorted video sequence are selected as reference videos.
[0141] The loss function is constructed as follows:
[0142] The total loss function is as follows:
[0143]
[0144] Used for optimized action segmentation in the action unit parser, the calculation method is as follows:
[0145] BCE(x,y)=-(x log y+(1-x)log(1-y))
[0146]
[0147] in, is the true probability of the hth action transition occurring in the tth frame; relatively, is the predicted probability of the zth action transition occurring in the tth frame; and The formula for the entire score evaluation framework is as follows:
[0148]
[0149] in, is the prediction score output by the model, X Q is the true score of the query video.
[0150] The following examples are used to verify the beneficial effects of the present invention:
[0151] Example 1:
[0152] The multi-level motion analysis method for motion quality assessment in this embodiment is specifically prepared according to the following steps:
[0153] The dataset used in the experiment is the FineDiving dataset, a publicly available dataset in the field. Table 1 shows the performance comparison of the present invention and the most advanced existing methods on the FineDiving dataset using the Spearman rank correlation coefficient, relative L2 distance, AIoU@0.5, and AIoU@0.75 indicators. The higher the Spearman rank correlation coefficient, AIoU@0.5, and AIoU@0.75, the better, while the lower the relative L2 distance, the better. It can be seen that the present invention achieves the best performance in all indicators. Figure 3 The results of the proposed method for different motion segmentation types on the FineDiving dataset are shown. The predicted indices are very close to the actual indices, and the segmentation logic is also consistent with the actual situation. The experimental results verify the effectiveness of the proposed multi-level motion analysis method for motion quality assessment.
[0154] Table 1 Performance index comparison table
[0155]
[0156] The present invention may also have many other embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art may make various corresponding changes and modifications based on the present invention, but these corresponding changes and modifications should all fall within the scope of protection of the claims attached to the present invention.
Claims
1. A multi-level motion analysis method for motion quality assessment, characterized by: Including training process and prediction process; The training process runs as follows: Step 1: Input a video whose score needs to be queried. A perfect comparison mechanism is used to select a reference video corresponding to the query video and form a video pair with the query video, and then input it into the action unit parser. Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them; Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively; Step 4: Use the custom weight scoring module to combine different features at different stages to perform regression calculations on the score differences, and use this to obtain the predicted score of the query video; Step 5: Construct the loss function and use the Nadam optimizer with a learning rate of 0.0001 and automatic backpropagation training with the help of cosine annealing training method; The prediction process runs as follows: Step 1: Input a video whose score needs to be queried. Use a multi-instance balanced voting mechanism to select a corresponding number of reference videos to form multiple video pairs with the query video. Input the multiple video pairs into the action unit parser in sequence. Step 2: Use the action unit parser to extract the skeleton information and action unit representation images of the corresponding video, and perform action segmentation and alignment on them; Step 3: Use the motion parser and condition parser to extract the motion features and condition features contained in the action unit image set and skeleton information set respectively; Step 4: Use the custom weight scoring module to perform regression calculations on the score differences based on the different features at different stages, and use this to obtain the predicted score of the query video in the video pair; Step 5: Average the multiple prediction scores obtained from multiple video pairs input, and the result is the final score of the predicted video.
2. A multi-level motion analysis method for motion quality assessment according to claim 1, characterized in that: The operation steps of the action unit parser include: Step 1: Use the pose extractor of the YOLOv11 architecture to extract the skeleton coordinate information and bounding box coordinate information of the main athlete in the corresponding video; Step 2: Fuse the bounding box coordinates. If the number of bounding boxes in each frame exceeds 1, all bounding boxes are fused. The upper left corner of the fused bounding box is the minimum value of the upper left corner coordinates of all bounding boxes, and the lower right corner coordinates is the maximum value of the lower right corner coordinates of all bounding boxes, so that it can cover the main athlete area as much as possible. Step 3: Generate action unit representation images for each frame of the video based on the skeleton coordinates and image resolution. Specifically, for each frame, select the shoulders, hips, knees, and ankles from the skeleton coordinates, one for each part, for a total of four pairs of coordinates. If a required coordinate does not exist, it is represented as a null value. The steps are as follows: Step 3.1: First, calculate the size of the action unit image. All action unit images are square, and their width and height are expressed as CutSize. The calculation method of CutSize is as follows: Among them, width box and height box are the width and height of the bounding box of the player area in the first frame of the sequence, width image and height image are the overall width and height of the frame image respectively; Step 3.2: Generate a black canvas with a width of 4 × CutSize pixels and a height of 2 × CutSize pixels. Divide it into two parts according to its width. In the left part, generate the region of interest images of the shoulder, hip, knee, and ankle from top to bottom and from left to right. Each region of interest is generated by intercepting a region of size CutSize × CutSize pixels centered on the average center coordinate of a pair of coordinates of the part. In the right part, intercept the entire bounding box area and resize the image to a rectangle with a width of 2 × CutSize pixels and a height of 2 × CutSize pixels. Step 3.3: Change the overall size of the stitched image to 128×128 pixels; Step 4: Use the bounding box coordinate information to perform rectangular cropping on the video image; Step 5: Input the cropped image sequence into a color feature analysis module 1 composed of an I3D network to obtain color features; Step 6: Input the skeleton information sequence corresponding to the image sequence described in step 5 into a skeleton sequence parsing module 1 composed of LSTM, Transformer-Encoder and Nystrom-Attention to obtain skeleton feature 1. The operation steps of the skeleton sequence parsing module are as follows: Step 6.1: Input the skeleton information sequence into Nystrom-Attention, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain the first-level skeleton feature X1; Step 6.2: Input the skeleton information sequence into LSTM, then input the output of LSTM into Nystrom-Attention, and finally input the output of Nystrom-Attention into Transformer-Encoder to obtain the secondary skeleton feature X2; Step 6.3: Concatenate the first-level skeleton feature X1 and the second-level skeleton feature X2 with the dimension 1 of the feature tensor to obtain the final skeleton feature 1; Step 7: Stack the color features and the final skeleton features in the time dimension and regress them into a probability matrix through a 3-layer simple MLP. The number of rows R of this probability matrix represents the division of a motion into R+1 stages, the number of columns C represents the total number of frames after video alignment, and the number in the rth row and cth column represents the probability of transitioning from the rth stage to the r+1th stage occurring in the cth frame. Step 8: Divide the skeleton information set into R+1 stages according to the probability matrix, divide the motion unit set into R+2 stages, which includes a conditional stage, which is the splash stage in diving, and align each stage.
3. A multi-level motion analysis method for motion quality assessment according to claim 2, characterized in that: The structure and operation steps of the motion analyzer are as follows: The structure contains: Color feature analysis module 2: consists of I3D network; Skeleton feature parsing module 2: It consists of three components: Transformer-Encoder, LSTM and Nystrom-Attention. Run steps: Step 1: Input the action units of the first R+1 stages, i.e., the set of action units excluding the conditional stage, into the second color feature analysis module to obtain color features; Step 2: Input the skeleton information set of the first R+1 stages, i.e., excluding the conditional stage, into the skeleton feature parser 2 to obtain the skeleton feature 2; The operating process in the skeleton feature analysis module 2 is as follows: Step 2.1: Input the skeleton information set into Nystrom-Attention, and then input the output of Nystrom-Attention into LSTM to obtain the primary skeleton features; Step 2.2: Input the primary skeleton features into Nystrom-Attention again, and then input the output of Nystrom-Attention into Transformer-Encoder to obtain skeleton features; Step 3: Stack and fuse the color feature 2 and the skeleton feature 2 in the time dimension to finally obtain the motion feature.
4. A multi-level motion analysis method for motion quality assessment according to claim 2, characterized in that: The structure and operation steps of the conditional parser are as follows: The structure includes: I3D network as the skeleton network. Running steps: Input the R+2th stage, that is, the set of action units in the conditional stage, to obtain the conditional features.
5. A multi-level motion analysis method for motion quality assessment according to claim 1, characterized in that: The structure and operation steps of the custom weight scoring module are as follows: The structure contains: Feature Difference Parsing Module: This is a ViT-Decoder that applies a cross-attention mechanism; Score regression module: contains a 3-layer simple MLP network; Custom weight score fusion module; Run steps: Step 1: Input the motion features obtained from the query video and the reference video into the feature difference analysis module to obtain the motion feature differences corresponding to the two videos; Step 2: Input the conditional features obtained from the query video and the reference video into the feature difference analysis module to obtain the conditional feature differences corresponding to the two videos; Step 3: Input the motion feature difference and condition feature difference into the score regression module to obtain the motion part score difference and condition part score difference respectively; Step 4: Add the score difference of the sports part and the score difference of the condition part according to the custom weight to obtain the total score difference; Step 5: Add the total score difference to the true score of the reference video to obtain the final prediction score of the query video.
6. A multi-level motion analysis method for motion quality assessment according to claim 1, characterized in that: The description of the perfect comparison mechanism is as follows: A multi-process variable ReferenceDict is introduced during the training process to maximize the generalization performance when performing action quality assessment; this variable is used to store a list of video names that have been compared with each query video for training; in each training cycle, videos that are not in the list, that is, videos that have not been compared with the query video, are selected for comparison training.
7. A multi-level motion analysis method for motion quality assessment according to claim 1, characterized in that: The multi-instance balanced voting mechanism is described as follows: For each prediction, 10 reference videos with the same action type as the query video are selected from the list of the user-defined training set for score prediction. The prediction scores obtained based on these 10 reference videos are averaged and the calculated average value is used as the final prediction score. The reference video selection strategy is as follows: first, all videos matching the user-specified query video are filtered out from the user-defined training set based on the action type; then, all filtered videos are sorted according to the absolute value of the original frame number difference between each filtered video and the user-specified query video; finally, the first 10 videos from the sorted video sequence are selected as reference videos.
8. A multi-level motion analysis method for motion quality assessment according to claim 1, characterized in that: The loss function is constructed as follows: The total loss function is as follows: Used for optimized action segmentation in the action unit parser, the calculation method is as follows: BCE(x,y)=-(x log y+(1-x)log(1-y)) in, is the true probability of the hth action transition occurring in the tth frame; relatively, is the predicted probability of the zth action transition occurring in the tth frame; and The formula for the entire score evaluation framework is as follows: in, is the prediction score output by the model, X Q is the true score of the query video.