Method and device for scoring movement quality of kicking movement and storage medium

Abstract

The invention discloses a skating motion quality scoring method, a computer device and a storage medium, which can pay attention to the technical performance of fine-grained key sub-motions so as to carry out accurate TES prediction, and can process long-sequence time sequence feature association learning facing skating motion quality evaluation so as to improve the accuracy of skating motion quality evaluation. According to the method, the effectiveness of long sequence feature expression is improved, and meanwhile, fine-grained positioning is performed on sub-actions in the sequence, so that key technical performance of the sub-actions is accurately estimated, and the action quality evaluation accuracy is improved; by using a time sequence interaction encoder in a scoring model, a time context relationship in long-sequence feature expression can be constructed when a score decoupling model performs an uncertainty regression process, and long-time-sequence context association information is incorporated into learned feature expression, so that the accuracy of feature expression is further improved, and the accuracy of the long-sequence feature expression is improved. And the PCS and TES score prediction accuracy is improved. The method is widely applied to the technical field of action quality evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of action quality evaluation, in particular to a method for scoring the quality of figure skating actions, a computer device and a storage medium. Background Art

[0002] The action quality assessment (AQA) task aims to design an intelligent scoring system that can automatically and objectively score competitive sports and various actions that need to be standardized. The action quality assessment task has a wide range of application scenarios, such as surgical skill rating and medical rehabilitation training in healthcare, athlete posture correction training systems in sports and artificial intelligence referee systems in competitions, operational compliance analysis and dangerous behavior monitoring in industrial production, etc. With the development of computer technology, supported by massive data and high-performance computing equipment, more and more video understanding and analysis methods have been proposed, and the action quality assessment task has also received more and more attention and application.

[0003] Figure skating, also known as figure skating, is a challenging long-sequence sport. The quality assessment of long-sequence movements not only requires scoring the performance of the entire sequence, but also requires fine-grained analysis of the key technical movements in the sequence and targeted quality assessment. In practical applications, the quality assessment of long-sequence movements involves more contextual information and diverse movement changes. For example, in competitive figure skating competitions, the movement sequences contain more complex and diverse technical sub-movement segments, and the quality assessment is more challenging. At present, there are problems in the quality scoring technology of figure skating movements, such as a lack of attention to fine-grained technical sub-movements. Summary of the invention

[0004] In view of the technical problems existing in the current figure skating movement quality scoring technology, such as the lack of attention to fine-grained technical sub-movements, the purpose of the present invention is to provide a figure skating movement quality scoring method, a computer device and a storage medium.

[0005] On the one hand, an embodiment of the present invention includes a method for scoring the quality of figure skating movements, the method comprising the following steps:

[0006] Obtaining a scoring model; the scoring model includes a score decoupling model, a temporal feature interaction encoding model, and a score uncertainty regression model;

[0007] Obtain sample videos of figure skating and the corresponding real scores of program content and technical elements;

[0008] Processing the sample video of the figure skating sport to obtain a technical sub-movement positioning result;

[0009] The figure skating sample video is processed using the fractional decoupling model to obtain a decoupling feature and a first loss value; the decoupling feature includes a feature characterizing program content and a feature characterizing technical elements;

[0010] According to the technical sub-action positioning result, the decoupling feature is temporally interactively encoded using the temporal feature interactive coding model to obtain program content temporal interaction features and technical sub-action temporal interaction features;

[0011] Using the score uncertainty regression model, the program content temporal interaction features and the technical sub-action temporal interaction features are processed to obtain a prediction score, a second loss value, and a third loss value;

[0012] The scoring model is trained according to the first loss value, the second loss value and the third loss value.

[0013] Furthermore, the processing of the figure skating sample video to obtain a technical sub-action positioning result includes:

[0014] Encoding the sample video of the figure skating sport using a video feature encoder to obtain video sequence features;

[0015] Using a weakly supervised sub-action localization model to identify the video sequence features, and determine the start time point and the end time point of the technical sub-action in the video sequence features;

[0016] According to the start time point and the end time point of the technical sub-action, the positioning result of the technical sub-action is determined.

[0017] Further, the using the fractional decoupling model to process the figure skating sample video to obtain a decoupling feature and a first loss value includes:

[0018] Processing the video sequence features using the fractional decoupling model to obtain feature expressions of corresponding program content and technical elements;

[0019] The calculation is done by the following formula:

[0020] Q c =ω q ·q c ,K=ω k F,V=ω v F.

[0021]

[0022] Among them, q c is the prototype matrix of the computational decoupling feature, F is the video sequence feature, ω q ,ωk and ω v is the mapping matrix used by the fractional decoupling model, is the scale factor, c∈{T,P}, F c is the decoupling feature;

[0023] The calculation is done by the following formula:

[0024] L t =max(D + -D - +Margin,0).

[0025]

[0026] in, is the i-th feature representing the program content in the decoupled features, is the jth feature representing the program content in the decoupled features, is the i-th feature characterizing the technical element in the decoupling feature, is the jth feature representing the technical element in the decoupled feature, dist() is the distance function, Margin is the margin parameter, L t is the first loss value.

[0027] Further, the temporal feature interactive encoding model includes a first cross-attention module and a second cross-attention module;

[0028] The step of performing temporal interaction coding on the decoupled features according to the technical sub-action positioning result using the temporal feature interaction coding model to obtain the program content temporal interaction features and the technical sub-action temporal interaction features comprises:

[0029] Performing forward step processing on the feature representing the program content in the decoupling feature to obtain a first forward moving content decoupling feature;

[0030] Performing a backward step processing on the feature representing the program content in the decoupling feature to obtain a first backward moving content decoupling feature;

[0031] The features representing the program content in the decoupled features and the first forward moving content decoupled features are input into the first cross-attention module for processing, and the features representing the program content in the decoupled features and the first backward moving content decoupled features are input into the second cross-attention module for processing. The processing results of the first cross-attention module and the processing results of the second cross-attention module are average pooled to obtain the program content temporal interaction features.

[0032] Further, the method of performing temporal interaction coding on the decoupled features according to the technical sub-action positioning result using the temporal feature interaction coding model to obtain program content temporal interaction features and technical sub-action temporal interaction features includes:

[0033] Screening the features representing the technical elements in the decoupling features according to the technical sub-action positioning results to obtain a key sub-action feature sequence;

[0034] Performing forward step processing on the key sub-action feature sequence to obtain a second forward moving technology decoupling feature;

[0035] Performing a backward step processing on the key sub-action feature sequence to obtain a second backward moving technology decoupling feature;

[0036] The key sub-action feature sequence and the second forward moving technology decoupling feature are input into the first cross-attention module for processing, and the key sub-action feature sequence and the second backward moving technology decoupling feature are input into the second cross-attention module for processing. The processing results of the first cross-attention module and the processing results of the second cross-attention module are average pooled to obtain the technical sub-action temporal interaction features.

[0037] Further, the fractional uncertainty regression model includes a first regressor and a second regressor;

[0038] The step of using the score uncertainty regression model to process the program content temporal interaction features and the technical sub-action temporal interaction features to obtain a prediction score, a second loss value, and a third loss value includes:

[0039] Inputting the program content temporal interaction feature into the first regressor to obtain a first regression output generated by the first regressor;

[0040] Inputting the technical sub-action temporal interaction feature into the second regressor to obtain a second regression output generated by the second regressor;

[0041] Inputting the program content temporal interaction feature into the second regressor to obtain a third regression output generated by the second regressor;

[0042] Inputting the technical sub-action temporal interaction feature into the first regressor to obtain a fourth regression output generated by the first regressor;

[0043] Determine the prediction score and the second loss value according to the first regression output and the second regression output;

[0044] The third loss value is determined according to the first regression output, the second regression output, the third regression output and the fourth regression output.

[0045] Further, the training of the scoring model according to the first loss value, the second loss value and the third loss value includes:

[0046] Training the score decoupling model according to the first loss value;

[0047] The temporal feature interaction encoding model and the fractional uncertainty regression model are trained according to the second loss value and the third loss value.

[0048] Furthermore, the figure skating movement quality scoring method also includes:

[0049] Get the figure skating videos to be rated;

[0050] The trained scoring model is used to process the figure skating video to be scored to obtain a scoring value.

[0051] On the other hand, an embodiment of the present invention also includes a computer device, including a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the figure skating movement quality scoring method in the embodiment.

[0052] On the other hand, an embodiment of the present invention further includes a computer-readable storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to execute the figure skating movement quality scoring method in the embodiment.

[0053] The beneficial effects of the present invention are as follows: the figure skating action quality scoring method in the embodiment can focus on the technical performance of fine-grained key sub-actions to perform accurate TES prediction by introducing a technical sub-action positioning operation to obtain a technical sub-action positioning result during the training of the scoring model; by using a score decoupling model in the scoring model, PCS and TES can be respectively subjected to uncertainty decoupling regression based on the assistance of sub-action sequence positioning, and can handle the long-sequence temporal feature association learning faced by the figure skating action quality evaluation to improve the effectiveness of the long-sequence feature expression; at the same time, the sub-actions in the sequence are fine-grainedly positioned, and the key technical performance of the sub-actions can be accurately estimated, thereby improving the accuracy of the action quality evaluation; by using a temporal interaction encoder in the scoring model, the temporal context relationship in the long-sequence feature expression can be constructed when the score decoupling model performs an uncertainty regression process, and the long-sequence context association information can be incorporated into the learned feature expression, thereby further improving the feature expression accuracy and improving the PCS and TES score prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 Schematic diagram of the steps of the method for scoring the quality of figure skating movements in the embodiment;

[0055] Figure 2 It is a schematic diagram of the structure of the long sequence fine-grained sub-action localization algorithm in the embodiment;

[0056] Figure 3 It is a schematic diagram of the principle of performing temporal interactive coding on the features representing the program content in the decoupled features in the embodiment;

[0057] Figure 4 It is a schematic diagram of the principle of performing temporal interactive coding on the features representing the technical elements in the decoupling features in the embodiment;

[0058] Figure 5 Schematic diagram of the principle of the uncertainty score uncertainty regression model in the embodiment. DETAILED DESCRIPTION

[0059] The scoring of figure skating competitions is mainly divided into two parts, namely, Program Component Score (PCS) and Technical Element Score (TES). PCS reflects the overall performance of the athlete in the entire skating sport, while TES focuses on the athlete's technical ability in multiple technical sub-movements. The scores of the two parts are combined to form the final score of the athlete.

[0060] In order to solve technical problems such as lack of attention to fine-grained sub-movements, we can introduce technical sub-movement positioning operations, extract sequence fragments of each technical sub-movement performed by athletes, and focus on the technical performance of fine-grained key sub-movements to make accurate TES predictions. At the same time, we build an uncertainty score decoupling network, and perform uncertainty decoupling regression on PCS and TES respectively to obtain more accurate quality assessment results. Through the uncertainty score decoupling method assisted by sub-movement sequence positioning, we can handle the long-sequence temporal feature association learning faced by figure skating action quality assessment to improve the effectiveness of long-sequence feature expression. At the same time, we fine-grainedly locate the sub-movements in the sequence to accurately estimate the key technical performance of the sub-movements and improve the accuracy of action quality assessment.

[0061] Since the TES evaluation in figure skating movement quality assessment mainly focuses on some specific technical sub-movements of athletes and evaluates each technical element separately, the problem of evaluating the technical performance of sub-movements in TES score prediction can be solved by designing a process of weakly supervised temporal sub-movement positioning, extracting sequence fragments of each technical sub-movement performed by the athlete, and focusing on the technical performance of fine-grained key sub-movements.

[0062] When evaluating scores for PCS and TES, we can design a process for decoupling uncertainty scores to perform score decoupling and uncertainty regression. The score decoupling process is to express differentiated features for accurate prediction of PCS and TES scores, and assist PCS and TES score estimation; uncertainty regression semantically enhances the two types of features for PCS and TES score prediction.

[0063] When performing the uncertainty regression process, a temporal interaction encoder can be used to construct the temporal contextual relationship in the long-sequence feature expression, and incorporate the long-time series contextual association information into the learned feature expression to further improve the accuracy of the feature expression and improve the prediction accuracy of the PCS and TES scores.

[0064] Based on the above principles, in this embodiment, a method for scoring the quality of figure skating movements is provided. Figure 1 ,The figure skating movement quality scoring method includes the following steps:

[0065] P1. Obtain the scoring model;

[0066] P2. Obtain sample videos of figure skating and the corresponding actual scores of program content and technical elements;

[0067] P3. Process the sample video of figure skating to obtain the positioning results of technical sub-movements;

[0068] P4. Use the fractional decoupling model to process the sample video of figure skating to obtain the decoupling feature and the first loss value;

[0069] P5. According to the positioning results of the technical sub-actions, the decoupled features are temporally interactively encoded using the temporal feature interactive coding model to obtain the temporal interactive features of the program content and the temporal interactive features of the technical sub-actions;

[0070] P6. Use the score uncertainty regression model to process the program content temporal interaction features and the technical sub-action temporal interaction features to obtain the prediction score, the second loss value and the third loss value;

[0071] P7. Train the scoring model based on the first loss value, the second loss value, and the third loss value.

[0072] In step P1, the scoring model includes a score decoupling model, a temporal feature interaction encoding model, and a score uncertainty regression model.

[0073] In step P2, N sample figure skating videos and the corresponding true score S of the program content of each sample figure skating video may be collected. P And the actual score of technical elements S TThe sample video of figure skating can be a video of a figure skater performing figure skating movements in a real competition or training, and the real score of the program content S P The score of the program content is PCS, and the actual score of the technical element is S. T It is the technical element score TES given by the scorer to the action.

[0074] In this embodiment, when executing step P3, that is, processing the sample video of figure skating to obtain the technical sub-movement positioning result, the following steps may be specifically performed:

[0075] P301. Encode the sample video of figure skating using a video sequence feature encoder to obtain video sequence features;

[0076] P302. Use the weakly supervised sub-action localization model to identify the video sequence features and determine the start and end time points of the technical sub-actions in the video sequence features;

[0077] P303. Determine the positioning result of the technical sub-action based on the starting time point and ending time point of the technical sub-action.

[0078] Since a complete figure skating sample video usually lasts for several minutes and contains thousands of frames, in order to learn the entire video features more completely, the video can be divided into short clips in chronological order, and pre-trained video sequence feature encoders such as C3D (Convolutional 3D), I3D (Inflated 3D Convnet), VST (Video Swin Transformer), etc. can be used to obtain the video sequence features in the figure skating sample video.

[0079] Specifically, in step P301, three encoders, namely C3D, I3D, and VST, are evaluated, and VST is finally decided to be used as the video sequence feature encoder for video sequence feature encoding. VST can capture temporal and spatial feature associations in a video frame sequence by introducing the time dimension. By setting a window attention mechanism, VST can capture long-distance dependencies while maintaining efficient calculation, thereby better understanding the video content.

[0080] In step P301, the sample video of the figure skating sport is input into the video sequence feature encoder VST for processing, and the video sequence feature output by the video sequence feature encoder VST is obtained. Each figure skating sample video is divided into M short clips, f m Represents the features of the mth short clip in the sample video of figure skating.

[0081] The prediction of the Technical Element Score (TES) is achieved by individually evaluating the score of each key technical sub-movement (6 in the short program and 10 in the free skating) and integrating them to obtain the TES score of the entire competition sequence. In order to obtain an accurate score estimate of the key technical sub-movements, in step P302, a technical sub-movement positioning method is applied to extract the time starting point of each technical sub-movement performed by the athlete, extract the short sequence fragments of the key technical sub-movements, and on this basis, the score of each key technical sub-movement is estimated and integrated, thereby improving the accuracy of TES prediction.

[0082] In step P302, the ACM-Net algorithm (a weakly supervised temporal sub-action localization algorithm) is introduced as a weakly supervised sub-action localization model to preliminarily determine the start and end time points of technical sub-actions in the figure skating video.

[0083] Specifically, the algorithm structure of the weakly supervised sub-action localization model is as follows: Figure 2 See Figure 2 Based on the extracted short segment features, the weakly supervised sub-action localization model designs an attention learning mechanism to learn the attention feature expression from the short segment feature sequence, and then combines the learned attention features to weight the sub-action category activation sequence, and finally further refines and determines the sub-action localization results through a multi-instance learning method. Combined with the score decoupling algorithm for PCS and TES score prediction in the present invention, the preliminary learned key technical sub-action localization results are optimized to obtain the accurate timing starting point of the technical sub-action.

[0084] In step P302, the video sequence features The input is processed into the weakly supervised sub-action localization model, which recognizes the characteristics of the video sequence. and mark the technical sub-actions in the video sequence features The start time point and end time point in .

[0085] In step P303, the start time point and the end time point of the technical sub-action identified in step P302 can be represented as the technical sub-action positioning result where a m ∈{0,1}. m = 0 means that there is no key technical sub-action or background in the mth short clip in the sample video of figure skating, a m =1 indicates that the key technical sub-action or background appears in the mth short clip in the sample video of figure skating.

[0086] By executing steps P301-P303, the technical sub-action positioning result is obtained. It can be used as guidance information to correctly predict PCS and TES in the score decoupling module. The prediction of TES is more dependent on key technical sub-actions, so the learned sub-action timing positioning helps to learn more accurate sub-action feature expressions to obtain accurate TES score predictions. The obtained technical sub-action positioning results Each sub-action start time proposal can be optimized to make it closer to the actual sub-action segmentation.

[0087] In steps P301-P303, the main purpose of segmenting short segments from long sequences is to extract video frame features and avoid using single-frame features. Since the length of the short segment in the video sequence is much smaller than the length of the sub-action, it does not affect the subsequent sub-action positioning.

[0088] With the goal of accurately predicting PCS and TES scores, the present invention uses a Transformer network model to perform decoupled learning to obtain feature representations guided by PCS and TES score estimation, respectively.

[0089] To refine the feature expression, in this embodiment, DETR (DEtection TRansformer) (a non-regression and non-mask version of Transformer) can be used as a score decoupling model to perform PCS and TES score decoupling.

[0090] Specifically, in this embodiment, for the video sequence feature The score decoupling model DETR is used to learn the feature expressions for PCS and TES score prediction.

[0091] To this end, in this embodiment, step P4 is performed, that is, the sample video of the figure skating sport is processed using the fractional decoupling model DETR to obtain the decoupling feature F c and the first loss value L t In this step, we first use the score decoupling model DETR to analyze the video sequence features. Processing is performed to obtain the prototype matrix q c , where c∈{T,P}. That is, the prototype matrix q c In fact, it includes q P and q T Two parts.

[0092] In step P4, the video sequence features are then and the prototype matrix q c Calculate the cross attention and get F c Specifically, it is calculated by the following formula:

[0093] Q c =ω q ·q c,K=ω k F,V=ω v F.

[0094]

[0095] Among them, ω q ,ω k and ω v The mapping matrix used by the score decoupling model DETR, is the scale factor, c∈{T,P}. When c=T, the above formula is actually the feature of the video sequence and the prototype matrix q T Calculate cross attention to obtain features that represent technical elements in the decoupled features When c = P, the above formula is actually the video sequence feature and the prototype matrix q P Calculate cross attention to obtain the features that represent the program content in the decoupled features Among them, the features that represent the program content in the decoupled features It is a PCS-oriented feature that decouples the features that represent the technical elements in the feature These two features corresponding to PCS and TES score estimation constitute the decoupled feature F c (c∈{T,P}).

[0096] Since the evaluation rules of PCS and TES are different, the feature expression for PCS is the feature representing the program content in the decoupled feature. Features that represent technical elements in decoupled features In order to learn the feature expressions that can distinguish the PCS score prediction orientation and the TES score prediction orientation, in step P4, the triplet loss (Triplet Loss) L t As the first loss value, we train the score decoupling model DETR. The first loss value is the triplet loss (Triplet Loss) L t The calculation formula is as follows:

[0097] L t =max(D + -D - +Margin,0).

[0098]

[0099] in, To decouple the features that characterize the program content The i-th feature in To decouple the features that characterize the program content The jth feature in To decouple the features of the technical elements The i-th feature in To decouple the features of the technical elements The jth feature in , since i≠j, and are different characteristics, and are also different features; dist() is the distance function, which can be L2 distance or other distance; Margin is the margin parameter, and Margin can be set to a suitable fixed value.

[0100] The first loss value is the triplet loss (Triplet Loss) L t In the calculation formula of , the feature representing the program content in the decoupled feature and decoupling features to characterize technical elements is used to form positive and negative sample pairs. In addition, the first loss value, namely the triplet loss (Triplet Loss) L t In the calculation formula of , difficult positive sample pairs and difficult negative sample pairs are also constructed to support a more robust training process.

[0101] The calculation formula of the first loss value used in step P4 is the triple loss L calculated t It helps the score decoupling model DETR to learn to converge the distribution of the same type of features of PCS and TES, and to diverge the distribution of different types of features between the two, so as to achieve goal-oriented learning of PCS and TES feature expression.

[0102] Since PCS and TES score predictions follow different scoring rules, in this embodiment, different temporal interaction coding is performed for the semantic feature extraction of the two. Among them, the steps of performing temporal interaction coding for PCS are P501A-P503A, and the corresponding process is as follows Figure 3 As shown; the steps for TES to perform time-series interactive coding are steps P501B-P504B, and the corresponding process is as follows Figure 4 By executing Figure 3 and Figure 4 The steps shown can integrate the temporal sequence context information and learn the sequence semantic feature expression.

[0103] In this embodiment, when executing step P5, that is, according to the technical sub-action positioning result Use the temporal feature interaction coding model to perform temporal interaction coding on the decoupled features to obtain the temporal interaction features of the program content and technical sub-action temporal interaction characteristics In this step, you can perform the following steps:

[0104] P501A. Features that characterize program content in decoupled features Perform forward movement processing to obtain the first forward movement content decoupling feature

[0105] P502A. Features that characterize program content in decoupled features Perform backward movement processing to obtain the first backward movement content decoupling feature

[0106] P503A. Decouple the features that characterize program content Decoupling features from the first forward moving content Input to the first cross attention module for processing, the features representing the program content in the decoupled features Decoupling features from the first backward movement content The data is input to the second cross attention module for processing, and the processing results of the first cross attention module and the second cross attention module are averaged and pooled to obtain the temporal interaction characteristics of the program content.

[0107] Steps P501A-P503A are to decouple the features that characterize the program content. The steps of performing sequential interactive coding are as follows. Figure 3 shown.

[0108] Reference Figure 3 In step P501A, the feature representing the program content in the decoupled feature Move forward 2 steps to obtain the first forward moving content decoupling feature In step P502A, the feature representing the program content in the decoupled feature Move backward 2 steps to obtain the first backward movement content decoupling feature

[0109] Reference Figure 3 In step P503A, the feature representing the program content in the decoupled feature Decoupling features from the first forward moving content Combined, and input into the first cross attention module in the temporal feature interactive coding model for processing; in step P504A, the features representing the program content in the decoupled features Decoupling features from the first backward movement content The first and second cross-attention modules are combined and input into the temporal feature interaction encoding model for processing; the first and second cross-attention modules learn the sequence feature expression of temporal interaction, and finally, the processing results of the first and second cross-attention modules are averaged and pooled to obtain the temporal interaction features of the program content. Program content temporal interaction characteristics Can be used for PCS score prediction.

[0110] In this embodiment, when executing step P5, that is, according to the technical sub-action positioning result Use the temporal feature interaction coding model to perform temporal interaction coding on the decoupled features to obtain the temporal interaction features of the program content and technical sub-action temporal interaction characteristics In this step, you can perform the following steps:

[0111] P501B. Positioning results based on technical sub-actions Characteristics of technical elements in decoupling features Screening to obtain key sub-action feature sequences

[0112] P502B. Key sub-action feature sequence Perform forward step processing to obtain the second forward moving technology decoupling feature

[0113] P503B. Key sub-action feature sequence Perform the backward movement process to obtain the second backward movement technology decoupling feature

[0114] P504B. Sequence the key sub-action features Decoupling features from the second forward movement technique Input to the first cross attention module for processing, the key sub-action feature sequence Decoupling features from the second backward movement technique The data is input to the second cross attention module for processing, and the processing results of the first cross attention module and the second cross attention module are averaged and pooled to obtain the technical sub-action temporal interaction features.

[0115] Steps P501B-P504B are to characterize the features of the technical elements in the decoupling features. Steps for performing sequential interactive coding. The principles of steps P501B-P504B are as follows Figure 4 shown.

[0116] In the traditional AQA algorithm framework, the TES score estimation regression (TES-R) module usually aggregates the features of the entire sequence, and the aggregated feature expression is used for TES score prediction. This approach often ignores the fact that TES score estimation in competitions usually scores the key sub-actions in the game sequence separately and integrates the final score. Therefore, step P501B improves this situation.

[0117] In step P501B, following the rules of the competition, first locate the result of the technical sub-action Features that characterize technical elements in decoupling features Information fusion is performed to enhance the feature expression of technical sub-actions. Under the guidance of According to the distribution of sub-movements in the sequence, they are divided into several sub-movement segments, which are unified into fixed lengths through sampling, interpolation and other processing for subsequent feature learning; since the number of technical sub-movements in each competition is fixed in figure skating competitions, for example, there are 6 jumps and rotations in the short program, 6 sub-movement segments and 5 background segments will be generated after segmentation (the two background segments at the beginning and end of the performance sequence are discarded here). Through screening processing, only the technical sub-movement segments are retained, thereby obtaining the key sub-movement feature sequence Here M A Indicates the number of technical sub-movements in a competition video. In figure skating short program, M A =6, M in free skating A =10.

[0118] In steps P502B-P504B, refer to Figure 4 , with the key sub-action feature sequence F T (m A ) as input, the first cross attention module and the second cross attention module in the temporal feature interaction encoding model are used to perform a cross-over on each key sub-action feature sequence F T (m A ) to learn the key sub-action feature sequence F T (m A ) is a semantic expression that combines related features. Specifically, refer to Figure 4 , similar to executing steps P501A-P503A to decouple the features representing the program content As in the case of temporal interactive encoding, the forward and backward shift strategies are still used to adjust the key sub-action feature sequence F T (m A), and take the shifted sequence as input, use the first cross attention module and the second cross attention module to perform feature sequence interaction encoding, and finally obtain the technical sub-action temporal interaction feature

[0119] By executing steps P501A-P503A and steps P501B-P504B, and using PCS and TES temporal feature interactive encoding, the contextual associations between temporal segments in an entire figure skating competition video can be explored to obtain the semantic representation of the continuity of the motion sequence, thereby better fitting the scoring rules of PCS and TES; using the Cross-Attention (CA) module for feature sequence interactive encoding can realize interactive learning of different temporal features.

[0120] In this embodiment, the score uncertainty regression model includes a first regressor (PCS-R) and a second regressor (TES-R). When executing step P6, that is, using the score uncertainty regression model to process the program content temporal interaction features and the technical sub-action temporal interaction features to obtain the predicted score, the second loss value and the third loss value, the following steps can be specifically performed:

[0121] P601. Interactive features of program content Inputting into the first regressor (PCS-R), obtaining a first regression output generated by the first regressor (PCS-R);

[0122] P602. The technical sub-action timing interaction characteristics Input to the second regressor (TES-R), and obtain the second regression output generated by the second regressor (TES-R);

[0123] P603. Interactive features of program content timing Input to the second regressor (TES-R), and obtain the third regression output generated by the second regressor (TES-R);

[0124] P604. The technical sub-action timing interaction characteristics inputting into the first regressor (PCS-R) to obtain a fourth regression output generated by the first regressor (PCS-R);

[0125] P605. Determine a prediction score and a second loss value according to the first regression output and the second regression output;

[0126] P606. Determine a third loss value based on the first regression output, the second regression output, the third regression output and the fourth regression output.

[0127] Steps P601-P606 are to combine PCS features and TES features and Mapping to motion score, that is, using regression algorithm to achieve the mapping between PCS and TES feature expressions decoupled from the sequence and the score. In this embodiment, a linear regressor R is used. c (·) to achieve score prediction. In order to increase the robustness of the score regression results, the present invention introduces uncertainty regression. The introduction of uncertainty here is to measure and control the instability of the output results under given input features.

[0128] If the traditional mean square error (MSE) loss is used for regression constraints, when the features are disturbed and deviate from the correct description of the sample, the output of the regression is also easily directly affected. In the present invention, although the ternary loss L is applied t To perform feature decoupling operations for PCS and TES score estimation on action sequences, the distribution of decoupled features is not completely independent, which may cause score prediction confusion. In order to obtain more robust prediction results, uncertainty regression is introduced and the parameter α is set c To control uncertainty.

[0129] Based on the above principles, refer to Figure 5 , a two-headed linear regressor (consisting of two fully connected layers (FC) and one output layer) is selected as the score uncertainty regression model, and PCS and TES score prediction regression are implemented. In the two-headed linear regressor, one of the heads performs score prediction and outputs the predicted score While the other head estimates the uncertainty parameter α c The value of .

[0130] According to the scoring rules of PCS and TES, the feature representations for PCS and TES should contain different semantic information. In other words, one type of feature should not provide a positive contribution to the score prediction of another type. In this embodiment, an uncertainty measurement function U(·) is set to quantify the deviation between the predicted score and the true score. In addition, in addition to using U(·) for positive uncertainty measurement, this implementation case also proposes a negative uncertainty regression measurement to enhance the independence and difference of the semantic feature representations of PCS and TES.

[0131] The negative uncertainty regression metric exchanges the feature inputs of the first regressor (PCS-R) and the second regressor (TES-R), and the regressor outputs a pseudo-prediction result, and then the uncertainty metric function U(·) quantifies the deviation between the pseudo-prediction score and the true score. Ideally, the positive uncertainty metric value should be close to zero, and the larger the negative uncertainty metric value, the more conducive it is to training the feature expression of PCS and TES with discrete distribution. This embodiment sets two weight parameters λ real ,λ fake, which are used to constrain the positive uncertainty measure and the negative uncertainty measure respectively.

[0132] Specifically, refer to Figure 5 In step P601, the program content timing interaction feature Input to the first regressor (PCS-R) to obtain the first regression output generated by the first regressor (PCS-R) in represents the first regressor (PCS-R) for the temporal interaction characteristics of program content The prediction score, α P represents the uncertain parameters output by the first regressor (PCS-R).

[0133] Reference Figure 5 In step P602, the technical sub-action temporal interaction feature Input to the second regressor (TES-R) to obtain the second regression output generated by the second regressor (TES-R) in represents the temporal interaction characteristics of the second regressor (TES-R) on the technical sub-actions The prediction score, α T represents the uncertain parameters of the output of the second regressor (TES-R).

[0134] In steps P601-P602, and The prediction scores that constitute the output of the fractional uncertainty regression model.

[0135] Reference Figure 5 In step P603, the program content timing interaction feature Input to the second regressor (TES-R) to obtain the third regression output generated by the second regressor (TES-R) Third regression output is a spurious regression output.

[0136] Reference Figure 5 In step P604, the technical sub-action temporal interaction feature Input to the first regressor (PCS-R) to obtain the fourth regression output generated by the first regressor (PCS-R) Fourth regression output is also a spurious regression output.

[0137] In step P605, and The prediction score is then constructed based on the first regression output and the second regression output And the program content real score S obtained in step P2 P and Technical Element True Score S T , calculate the second loss value L u In this embodiment, the second loss value L u The calculation formula is:

[0138]

[0139] The second loss value l u is an uncertainty regression loss, where U(·) is the uncertainty measure function, which is calculated as:

[0140]

[0141] Where S c Indicates the true score, for example, when c = P, S c Indicates the real score of the program content S P , c = T when S c Indicates the actual score of the technical element S T U(·) can be used to quantify the regression error in the case of pseudo regression and to train the model by minimizing all regression errors.

[0142] In order to ensure the correct regression of the two evaluation scores, a third loss value L is further designed in this embodiment. con The third loss value L con is a confusion loss, using triplet loss to reasonably distinguish positive regression from negative regression. In this embodiment, the third loss value L con That is, the calculation formula for confusion loss is:

[0143]

[0144] Among them, margin c is a parameter used to control the deviation between positive regression and negative regression. P , α T and Can be output from the first regression Fourth regression output Second regression output And the third regression output Obtained from.

[0145] In summary, in this embodiment, the total loss function used to train the scoring model is shown in the following formula.

[0146] L all =λ1·L t +λ2·L con +λ3·Lu

[0147] Among them, L t is the first loss value, L u is the second loss value, L con is the third loss value, λ1,λ2,λ3 are weight parameters.

[0148] In step P7, according to the first loss value L t , the second loss value L u and the third loss value L con , train the scoring model. Specifically, according to the first loss value L t The size of the second loss value L is used to adjust the parameters of the score decoupling model DETR, thereby training the score decoupling model DETR; u and the third loss value L con The size of is used to adjust the parameters of the first cross-attention module and the second cross-attention module in the temporal feature interaction encoding model, and the parameters of the first regressor and the second regressor in the score uncertainty regression model, so as to train the temporal feature interaction encoding model and the score uncertainty regression model.

[0149] In this embodiment, steps P1-P7 are steps of one round of training process. Generally, the training process of the scoring model includes multiple rounds of steps P1-P7. After executing multiple rounds of steps P1-P7 until the total loss function converges or the total number of rounds reaches the upper limit, the training of the scoring model is terminated.

[0150] After training the scoring model, you can perform the following steps:

[0151] S1. Obtain a figure skating video to be rated;

[0152] S2. Use the trained scoring model to process the figure skating video to be scored to obtain a scoring score.

[0153] In step S1, a figure skating video to be scored can be obtained by shooting at the competition site. When executing step S2, the figure skating video to be scored is equivalent to the figure skating sample video in step P2, and the video sequence feature encoder VST processes the figure skating video to be scored to obtain the technical sub-action positioning result, and uses the trained score decoupling model DETR to process the figure skating sample video to obtain the features representing the program content in the decoupling features and the features representing the technical elements in the decoupling features, and according to the technical sub-action positioning result, uses the trained temporal feature interaction coding model to perform temporal interaction coding on the features representing the program content in the decoupling features and the features representing the technical elements in the decoupling features, and obtains the program content temporal interaction features and the technical sub-action temporal interaction features, and uses the trained score uncertainty regression model to process the program content temporal interaction features and the technical sub-action temporal interaction features, and obtains the scoring score, wherein the scoring score is equivalent to the predicted score obtained in step P6. The scoring score can represent the score obtained by the scoring model predicting the figure skating video to be scored, which specifically represents the score of the figure skating video to be scored in terms of program content and technical elements.

[0154] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the trained scoring model in this embodiment. In this experimental verification, the figure skating video data sets FineFS and Fis-V in competitive sports are used, and 6 comparison methods are selected for comparison experiments to statistically evaluate the accuracy. The 6 comparison methods are C3D+SVR (3D Convolutional neural networks+Support Vector Regression, three-dimensional convolutional neural network + support vector regression) algorithm, TSA-Net (Tube Self-Attention Network, Tube Self-Attention Network) algorithm, GDLT (Grade Decoupling for Long-term Action Assessment, Grade Decoupling for Long-term Action Assessment) algorithm, MS-LSTM (Multi-scale Convolutional Skip LSTM, Multi-scale Convolutional Long Short-term Memory Artificial Neural Network) algorithm, TSA (Temporal Segmentation Attention, Time Segmentation Attention) algorithm and ACTION-NET (dynAmic-static Context-aware attenTION NETwork, dynamic-static context-aware attention network) algorithm.

[0155] Table 1 shows the comparison of PCS and TES score prediction accuracy of the scoring model and the comparison method in this embodiment on the figure skating short program competitive sports video dataset FineFS.

[0156] Table 1

[0157]

[0158] Table 2 shows the comparison of PCS and TES score prediction accuracy of the scoring model and the comparison method in this embodiment on the figure skating short program competitive sports video dataset Fis-V.

[0159] Table 2

[0160]

[0161]

[0162] As shown in Table 1 and Table 2, the scoring model in this embodiment has obvious advantages over these algorithms in PCS and TES prediction. The experimental results show that technical action positioning has a significant improvement in TES prediction. In addition, the scoring model in this embodiment improves the two score predictions of the dataset FineFS by an average of about 0.037, which proves the robustness and effectiveness of the temporal feature interaction encoding model and uncertainty regression used by the scoring model in this embodiment.

[0163] A computer program for executing the figure skating movement quality scoring method in this embodiment can be written and written into a computer device or a storage medium. When the computer program is read out and run, the figure skating movement quality scoring method in this embodiment is executed, thereby achieving the same technical effect as the figure skating movement quality scoring method in the embodiment.

[0164] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. In addition, the descriptions of up, down, left, right, etc. used in the present disclosure are only relative to the relative positional relationship of the components of the present disclosure in the accompanying drawings. The singular forms of "a", "" and "the" used in the present disclosure are also intended to include the plural forms, unless the context clearly indicates other meanings. In addition, unless otherwise defined, all technical and scientific terms used in this embodiment have the same meaning as those generally understood by those skilled in the art. The terms used in the specification of this embodiment are only for describing specific embodiments and are not intended to limit the present invention. The term "and / or" used in this embodiment includes any combination of one or more related listed items.

[0165] It should be understood that, although the term first, second, third etc. may be adopted to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish the same type of elements from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as" etc.) provided by the present embodiment is only intended to better illustrate embodiments of the present invention, and unless otherwise required, the scope of the present invention will not be limited.

[0166] It should be appreciated that embodiments of the present invention may be implemented or enforced by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method may be implemented in a computer program using standard programming techniques - including a non-transitory computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer to operate in a specific and predefined manner - according to the methods and drawings described in the specific embodiments. Each program may be implemented in a high-level procedural or object-oriented programming language to communicate with a computer system. However, if desired, the program may be implemented in assembly or machine language. In any case, the language may be a compiled or interpreted language. In addition, the program may be run on a programmed dedicated integrated circuit for this purpose.

[0167] In addition, the operations of the process described in this embodiment may be performed in any suitable order, unless otherwise indicated in this embodiment or otherwise clearly contradicted by the context. The process described in this embodiment (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed on one or more processors in common, by hardware or a combination thereof. A computer program includes a plurality of instructions that may be executed by one or more processors.

[0168] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or part thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the above steps in conjunction with a microprocessor or other data processor, the invention of this embodiment includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention also includes the computer itself.

[0169] The computer program can be applied to input data to perform the functions of the present embodiment, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0170] The above are only preferred embodiments of the present invention. The present invention is not limited to the above embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical solutions and / or implementation methods may have various modifications and changes.

Claims

1. A method for scoring the quality of figure skating movements, characterized in that: The figure skating movement quality scoring method comprises: Obtaining a scoring model; the scoring model includes a score decoupling model, a temporal feature interaction encoding model, and a score uncertainty regression model; Obtain sample videos of figure skating and the corresponding real scores of program content and technical elements; Processing the sample video of the figure skating sport to obtain a technical sub-movement positioning result; The figure skating sample video is processed using the fractional decoupling model to obtain a decoupling feature and a first loss value; the decoupling feature includes a feature characterizing program content and a feature characterizing technical elements; According to the technical sub-action positioning result, the decoupling feature is temporally interactively encoded using the temporal feature interactive coding model to obtain program content temporal interaction features and technical sub-action temporal interaction features; Using the score uncertainty regression model, the program content temporal interaction features and the technical sub-action temporal interaction features are processed to obtain a prediction score, a second loss value, and a third loss value; The scoring model is trained according to the first loss value, the second loss value and the third loss value.

2. The figure skating action quality scoring method according to claim 1, characterized in that: The processing of the sample video of the figure skating sport to obtain the technical sub-movement positioning result includes: Encoding the sample video of the figure skating sport using a video feature encoder to obtain video sequence features; Using a weakly supervised sub-action localization model to identify the video sequence features, and determine the start time point and the end time point of the technical sub-action in the video sequence features; According to the start time point and the end time point of the technical sub-action, the positioning result of the technical sub-action is determined.

3. The figure skating action quality scoring method according to claim 2, characterized in that: The step of processing the sample figure skating video using the fractional decoupling model to obtain a decoupling feature and a first loss value includes: Processing the video sequence features using the fractional decoupling model to obtain feature expressions of corresponding program content and technical elements; The calculation is done by the following formula: Q c =ω q ·q c ,K=ω k ·F,V=ω v ·F. Among them, q c is the prototype matrix of the computational decoupling feature, F is the video sequence feature, ω q ,ω k and ω v is the mapping matrix used by the fractional decoupling model, is the scale factor, c∈{T,P}, F c is the decoupling feature; The calculation is done by the following formula: L t =max(D + -D - +Margin,0). in, is the i-th feature representing the program content in the decoupled features, is the jth feature representing the program content in the decoupled features, is the i-th feature characterizing the technical element in the decoupling feature, is the jth feature representing the technical element in the decoupled feature, dist() is the distance function, Margin is the margin parameter, L t is the first loss value.

4. The method for scoring the quality of figure skating movements according to claim 1, characterized in that: The temporal feature interactive encoding model includes a first cross-attention module and a second cross-attention module; The step of performing temporal interaction coding on the decoupled features according to the technical sub-action positioning result using the temporal feature interaction coding model to obtain the program content temporal interaction features and the technical sub-action temporal interaction features comprises: Performing forward step processing on the feature representing the program content in the decoupling feature to obtain a first forward moving content decoupling feature; Performing a backward step processing on the feature representing the program content in the decoupling feature to obtain a first backward moving content decoupling feature; The features representing the program content in the decoupled features and the first forward moving content decoupled features are input into the first cross-attention module for processing, and the features representing the program content in the decoupled features and the first backward moving content decoupled features are input into the second cross-attention module for processing. The processing results of the first cross-attention module and the processing results of the second cross-attention module are average pooled to obtain the program content temporal interaction features.

5. The figure skating action quality scoring method according to claim 4, characterized in that: The step of performing temporal interaction coding on the decoupled features according to the technical sub-action positioning result using the temporal feature interaction coding model to obtain the program content temporal interaction features and the technical sub-action temporal interaction features comprises: Screening the features representing the technical elements in the decoupling features according to the technical sub-action positioning results to obtain a key sub-action feature sequence; Performing forward step processing on the key sub-action feature sequence to obtain a second forward moving technology decoupling feature; Performing a backward step processing on the key sub-action feature sequence to obtain a second backward moving technology decoupling feature; The key sub-action feature sequence and the second forward moving technology decoupling feature are input into the first cross-attention module for processing, and the key sub-action feature sequence and the second backward moving technology decoupling feature are input into the second cross-attention module for processing. The processing results of the first cross-attention module and the processing results of the second cross-attention module are average pooled to obtain the technical sub-action temporal interaction features.

6. The figure skating movement quality scoring method according to claim 1, characterized in that: The fractional uncertainty regression model includes a first regressor and a second regressor; The step of using the score uncertainty regression model to process the program content temporal interaction features and the technical sub-action temporal interaction features to obtain a prediction score, a second loss value, and a third loss value includes: Inputting the program content temporal interaction feature into the first regressor to obtain a first regression output generated by the first regressor; Inputting the technical sub-action temporal interaction feature into the second regressor to obtain a second regression output generated by the second regressor; Inputting the program content temporal interaction feature into the second regressor to obtain a third regression output generated by the second regressor; Inputting the technical sub-action temporal interaction feature into the first regressor to obtain a fourth regression output generated by the first regressor; Determine the prediction score and the second loss value according to the first regression output and the second regression output; The third loss value is determined according to the first regression output, the second regression output, the third regression output and the fourth regression output.

7. The figure skating action quality scoring method according to claim 1, characterized in that: The step of training the scoring model according to the first loss value, the second loss value, and the third loss value includes: Training the score decoupling model according to the first loss value; The temporal feature interaction encoding model and the fractional uncertainty regression model are trained according to the second loss value and the third loss value.

8. The method for scoring the quality of figure skating movements according to any one of claims 1 to 6, characterized in that: The figure skating action quality scoring method further includes: Get the figure skating videos to be rated; The trained scoring model is used to process the figure skating video to be scored to obtain a scoring value.

9. A computer device, characterized in that: It comprises a memory and a processor, the memory is used to store at least one program, and the processor is used to load at least one program to execute the figure skating movement quality scoring method according to any one of claims 1-8.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to execute the figure skating movement quality scoring method described in any one of claims 1-8 when executed by the processor.

Images