A method and device for action quality evaluation based on foreground enhancement

Abstract

The application discloses a motion quality evaluation method based on foreground enhancement, comprising the following steps: acquiring a video data set containing example videos and videos to be evaluated, and corresponding motion quality evaluation scores; constructing an initial model, including a feature extraction module, a spatial motion coding module, a temporal motion analysis module and a prediction module; and training the initial model by using the video data set to obtain an evaluation model for evaluating the motion quality in the video data. The application also provides a motion quality evaluation device. The method provided by the application can realize fine-grained motion quality evaluation, and has high accuracy and stability in sports scoring and other scenes.

Description

Technical Field

[0001] This invention belongs to the field of video quality assessment technology, and in particular relates to a method and apparatus for motion quality assessment based on foreground enhancement. Background Technology

[0002] Action quality assessment is a key task in video understanding, aiming to automatically score the quality of action execution. Existing methods mostly rely on overall scoring based on video-level features, neglecting the interaction between temporal, spatial, and semantic structures—especially the integration of skeleton information, which is crucial for capturing the complex dynamic kinematic features of human movements. This leads to insufficient scoring accuracy in complex sports such as diving and gymnastics.

[0003] Existing motion quality evaluation methods suffer from two main problems: firstly, insufficient spatial modeling, lacking precise modeling of the human skeletal structure, making it difficult to capture key motion areas; and secondly, coarse temporal analysis, failing to decompose motion sequences into semantically coherent steps, making it difficult to simulate the fine-grained scoring process of judges.

[0004] Patent document CN121188440A discloses a method, device, medium, and product for quality assessment and verification of embodied intelligence multi-source data. The method includes: converting embodied intelligence multi-source data into standardized format data; performing quality assessment on the standardized format data, including at least one of integrity check, usability check, and consistency check; loading a robot URDF model corresponding to the standardized format data, driving the URDF model to move according to joint data, and synchronously playing the corresponding visual data to achieve linkage playback of model actions and visual images; analyzing the linkage playback through a first major model and / or a second major model; wherein, the first major model is used to determine the physical safety of robot actions, and the second major model is used to determine the high-level semantic correctness of robot actions and task annotations.

[0005] Patent document CN120894373A discloses an intelligent recognition and evaluation method and system for assembly actions based on image sequences, including acquiring assembly image sequences, calculating three-dimensional spatial information, constructing a workpiece state transition diagram, predicting workpiece pose, generating standard action trajectories, comparing the deviation between the actual and standard trajectories, calculating action coordination indicators, judging the assembly state, analyzing action defects, and realizing assembly quality evaluation. Summary of the Invention

[0006] The purpose of this invention is to provide a motion quality evaluation method and apparatus based on foreground enhancement. This method integrates skeleton information to achieve fine-grained motion quality evaluation.

[0007] To achieve the first objective of this invention, the following technical solution is provided: a motion quality evaluation method based on foreground enhancement, comprising the following steps: Obtain the video dataset, which contains example videos and videos to be evaluated, along with their corresponding motion quality assessment scores; Build an initial model, including a feature extraction module, a spatial action encoding module, a temporal action parsing module, and a prediction module; The feature extraction module is used to extract video segment features from video data and construct the corresponding basic spatial features; The spatial motion coding module performs motion foreground masking on the input video data to obtain a basic attention map, and then fuses the basic attention map with the 2D skeleton key point map of the target unit in the video data to generate an enhanced spatial attention map. Based on the spatial attention map and basic spatial features, feature weighting calculation is performed to generate the corresponding spatial action code; The temporal action parsing module, based on the spatial action encoding, decomposes the video data using a DETR-type temporal decoder to obtain a temporal segmentation representation of multiple semantic steps; The InfoGCN algorithm is used to transform the 2D skeleton keypoint map of the target unit in the video data into an implicit topological structure. Attention mechanisms are applied to temporal segmentation representations and implicit topological structures of video data to obtain temporal encoding; The prediction module outputs the corresponding prediction score based on the time encoding of the example video and the video to be evaluated in the video data. The initial model is trained using a video dataset to obtain an evaluation model for assessing the quality of motion in video data.

[0008] This invention improves the scoring accuracy and stability in complex motion scenarios by fusing mask and 2D skeleton information and performing fine-grained analysis of spatiotemporal steps.

[0009] Specifically, the basic spatial feature sequence is extracted using a pre-trained I3D network, and its expression is as follows: ; in, This represents the input video data. Representing basic spatial characteristics, This represents the first and second layers of the I3D network.

[0010] Specifically, the expression for the spatial attention map is as follows: ; in, Indicates learnable parameters, This represents a 2D skeleton keypoint diagram. This represents the basic attention map.

[0011] Specifically, the expression for the spatial action encoding is as follows: ; in, This represents the third and fourth layers of the I3D network. Representing a spatial attention map, It represents the basic spatial characteristics.

[0012] Specifically, the temporal encoding is obtained by fusing temporal segmentation representation and implicit topological structure using a cross-attention mechanism.

[0013] Specifically, the expression for the cross-attention mechanism is as follows: ;in, Represents spatial action encoding, This represents a 2D skeleton keypoint diagram. ( ) represents a DETR-style time decoder. () represents the InfoGCN algorithm.

[0014] Specifically, the prediction module is constructed using a contrastive regression calculator, which calculates the score difference between the time-coded example video and the time-coded video to be evaluated to obtain the corresponding predicted score as the motion quality evaluation result of the video to be evaluated compared to the example video.

[0015] To achieve the second objective of this invention, the following technical solution is provided: a motion quality evaluation device for performing the steps of the above-described motion quality evaluation method based on foreground enhancement.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: By employing a spatial attention mechanism guided by a 2D skeleton and a mask, the foreground action area of ​​the human body is accurately located; by using a temporal parser guided by a 2D skeleton, the action is decomposed into semantically coherent steps, simulating the human scoring process and improving the ability to evaluate actions. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the motion quality evaluation method based on foreground enhancement provided in this embodiment; Figure 2 This embodiment provides a structural diagram of a mask and a 2D skeleton-guided spatial motion encoder and a 2D skeleton-guided temporal motion parser. Figure 3 This is a structural diagram of the contrast regressor provided in this embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] like Figure 1 As shown in this embodiment, a motion quality evaluation method based on foreground enhancement is provided, and its specific steps are as follows: Obtain the video dataset, which contains example videos and videos to be evaluated, along with their corresponding motion quality assessment scores; Build an initial model, including a feature extraction module, a spatial action encoding module, a temporal action parsing module, and a prediction module; The feature extraction module is used to extract video segment features from video data and construct the corresponding basic spatial features; The spatial motion coding module performs motion foreground masking on the input video data to obtain a basic attention map, and then fuses the basic attention map with the 2D skeleton key point map of the target unit in the video data to generate an enhanced spatial attention map. Based on the spatial attention map and basic spatial features, feature weighting calculation is performed to generate the corresponding spatial action code; The temporal action parsing module, based on the spatial action encoding, decomposes the video data using a DETR-type temporal decoder to obtain a temporal segmentation representation of multiple semantic steps; The InfoGCN algorithm is used to transform the 2D skeleton keypoint map of the target unit in the video data into an implicit topological structure. Attention mechanisms are applied to temporal segmentation representations and implicit topological structures of video data to obtain temporal encoding; The prediction module outputs the corresponding prediction score based on the time encoding of the example video and the video to be evaluated in the video data. The initial model is trained using a video dataset to obtain an evaluation model for assessing the quality of motion in video data.

[0020] More specifically, (1) data preparation and feature extraction; like Figure 2 As shown, for a given video input X, the video is first preprocessed. Specifically, the original video is sampled at a uniform frame rate and adjusted into fixed-length video segments.

[0021] For videos that are too short, repeated sampling and frame interpolation are used to pad the length. For videos that exceed a set length, sliding window sampling can be used to obtain a fixed number of video inputs. Subsequently, each video segment is adjusted to a uniform spatial resolution and normalized for input into the subsequent feature extraction network.

[0022] Then, the first and second layers of the I3D network, pre-trained on the Kinetics dataset, were used to extract features from the video segments. Edges, textures, local motion, and preliminary spatiotemporal variation information in the video segments were extracted through low-level convolutional structures to generate basic spatial features. :

[0023] This basic spatial feature preserves local action regions, human pose changes, and background context information in the video frame, providing a foundational input for subsequent foreground enhancement and spatial motion encoding.

[0024] (2) Spatial motion coding guided by 2D skeleton; Next, this embodiment introduces an action foreground enhancement mechanism between the second and third layers of the I3D network. Specifically, to incorporate prior knowledge of human structure, this embodiment also extracts human action masks and detects key points of the 2D human skeleton for target units in the input video. The action foreground mask is used to identify foreground regions in the video frame related to human action execution, such as human contour regions, limb movement regions, and local regions directly related to action quality. The action foreground mask can be obtained through human segmentation using YOLOv5. Subsequently, the action foreground mask is input into the convolutional network to generate a basic attention map. Simultaneously, by using HRNet to extract the human joint positions in each frame, such as key points for the head, shoulders, elbows, wrists, hips, knees, and ankles, the key point positions are mapped to a 2D skeleton keypoint map. The basic attention map is placed between the second and third layers of the I3D network. and the 2D skeleton key point map The resulting spatial attention map is obtained by fusion. .

[0025]

[0026] in These are learnable parameters that adaptively find the optimal weighting ratio as the model updates.

[0027] Obtain the enhanced spatial attention map Then, it is applied to basic spatial features. The spatial attention map is then spatially weighted. Specifically, positions with higher weights in the spatial attention map correspond to key regions of action execution, and the model enhances the feature responses of these regions; positions with lower weights typically correspond to background or action-irrelevant regions, and the model reduces their feature contribution. These weighted features are then fed into the third and fourth layers of the I3D network to further extract higher-level action semantic features, ultimately generating a 2D skeleton-guided spatial action code. .

[0028]

[0029] (3) Analysis of time-based motion guided by 2D skeleton; Based on the spatial action encoding obtained in step (2), a DETR-style temporal decoder is used. This DETR-style temporal decoder includes multiple learnable temporal query vectors, each used to adaptively aggregate a potential action semantic step from the video feature sequence. Through attentional interaction between the query vectors and the video temporal features, the temporal decoder can automatically learn key stage boundaries and stage feature representations in the action sequence, thereby decomposing the video features into temporal segmentation representations of multiple semantic steps. .

[0030] Meanwhile, this embodiment uses InfoGCN to characterize the 2D skeleton as an implicit topological structure between joints based on the spatial connection relationships and motion correlations between key points in the human body. This implicit topology not only includes fixed human skeletal connections, but also reflects the dynamic coordination between joints during different movements, such as the relationship between upper limb swing and trunk posture, and the relationship between lower limb exertion and changes in the body's center of gravity.

[0031] Finally, through the six-head cross-attention mechanism, and Fusion enhances the original features into temporal codes. To improve time consistency, the specific process can be represented by the following formula:

[0032] (4) such as Figure 3 As shown, this is the comparative regression provided in this embodiment; The query video features are calculated using a contrastive regressor. Example of a perfect score The difference between the scores is used to calculate the final predicted score.

[0033] in, This indicates the predicted score of the queried video. Indicates the score of the example video. This indicates the number of time-series segmentation in step three. This represents the difference in scores for each segment. By inputting into the 5-layer fully connected layer and Finally, the mean squared error between the predicted score and the true score, plus the cosine similarity of the prototype, is used as the optimization loss function for the entire model:

[0034] This embodiment also provides a motion quality evaluation device for performing the steps of the motion quality evaluation method based on foreground enhancement provided in the above embodiment.

[0035] The experimental results below show that, compared with existing motion instruction evaluation methods, the present invention can generate more accurate and stable predicted motion scores, achieving the best performance to date.

[0036] This embodiment is based on experiments using the FineDiving dataset, proposed in the paper "FineDiving: AFine-grained Dataset for Procedure-aware Action Quality Assessment" (authors Xu, Jinglin and Rao, Yongming and Yu, Xumin and Chen, Guangyi and Zhou, Jie and Lu, Jiwen, published in 2022), which includes 52 different action types. We compare this invention with the following three existing action instruction evaluation methods in our experiments: Existing Method 1: The method in the paper "FineDiving: A Fine-Grained Dataset for Evaluating the Quality of Program-Aware Actions" (authors Xu Jinglin, Rao Yongming, Yu Xumin, Chen Guangyi, Zhou Jie, and Lu Jiwen, published at the 2022 IEEE Conference on Computer Vision and Pattern Recognition) extracts spatiotemporal visual features through the I3D network and enhances the spatiotemporal features using a temporal segmentation attention module.

[0037] Existing Method Two: The method in the paper "RICA2: A Scoring Criterion-Informed and Calibrated Assessment of Action Quality" (authors Majeedi Abrar, Gajjala Viswanatha Reddy, GNVV Satya Sai Srinath Namburi, and Li Yin, published at the 2024 European Conference on Computer Vision) is based on the random embedding of action steps defined by a graph structure. This embedding method diffuses the probability density in the latent space, allowing the model uncertainty to be characterized. The graph-structured encoding scoring criteria can be inversely derived to obtain the quality score.

[0038] The third existing method is the one in the paper "FineCausal: A Causal-Based Interpretable Fine-Grained Action Quality Evaluation Framework" (authors Han Ruisheng, Zhou Kanglei, Atapour-Abarghouei Amir, Liang Xiaohui, and ShumHubert PH, published at the 2025 IEEE Conference on Computer Vision and Pattern Recognition Workshop). This method effectively separates human-centric foreground cues from background interference factors through a causal intervention module constructed using a graph attention network, and combines it with a temporal causal attention module to capture the fine temporal dependencies of each stage of the action.

[0039] The experiment in this embodiment uses Spearman's rank correlation coefficient. ) and relative Distance (the relative) distance, The predictive performance of motion quality evaluation is measured using indicators. Specifically, The metric is used to evaluate the accuracy of the model in predicting scores, and is calculated as follows:

[0040] and The stability of the model's predicted scores is then measured based on the highest score for each action. and lowest score calculate:

[0041] in and Let these represent the true score and the predicted score of the i-th sample, respectively. The larger the value and The smaller the value, the better the performance of the motion quality evaluation method.

[0042] Table 1

[0043] As shown in Table 1, this invention achieves better score prediction results. Existing motion quality evaluation methods only use spatiotemporal representation methods, which are difficult to take into account the kinematic features of the human skeleton; while this invention uses a joint representation of skeleton dynamic perception and latent prototype comparison to enhance the model's ability to model the contextual geometric structure and align the spatiotemporal-semantic feature distribution, thereby generating more accurate and realistic motion quality evaluations.

[0044] Furthermore, the terms "upper," "lower," "inner," "outer," "front," and "rear" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0045] Of course, the above description is only a specific embodiment of the present invention and is not intended to limit the scope of the present invention. All equivalent changes or modifications made to the structure, features and principles described in the claims of the present invention should be included in the scope of the claims of the present invention.

[0046] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A motion quality evaluation method based on foreground enhancement, characterized in that, Includes the following steps: Obtain the video dataset, which contains example videos and videos to be evaluated, along with their corresponding motion quality assessment scores; Build an initial model, including a feature extraction module, a spatial action encoding module, a temporal action parsing module, and a prediction module; The feature extraction module is used to extract video segment features from video data and construct the corresponding basic spatial features; The spatial motion coding module performs motion foreground masking on the input video data to obtain a basic attention map, and then fuses the basic attention map with the 2D skeleton key point map of the target unit in the video data to generate an enhanced spatial attention map. Based on the spatial attention map and basic spatial features, feature weighting calculation is performed to generate the corresponding spatial action code; The temporal action parsing module decomposes the video data based on the spatial action encoding to obtain a temporal segmentation representation of multiple semantic steps; Transform the 2D skeleton keypoint map of the target unit in the video data into an implicit topological structure; Attention mechanisms are applied to temporal segmentation representations and implicit topological structures of video data to obtain temporal encoding; The prediction module outputs the corresponding prediction score based on the time encoding of the example video and the video to be evaluated in the video data. The initial model is trained using a video dataset to obtain an evaluation model for assessing the quality of motion in video data.

2. The motion quality evaluation method based on foreground enhancement according to claim 1, characterized in that, The basic spatial feature sequence is extracted using a pre-trained I3D network, and its expression is as follows: ;in, This represents the input video data. Representing basic spatial characteristics, This represents the first and second layers of the I3D network.

3. The motion quality evaluation method based on foreground enhancement according to claim 1, characterized in that, The expression for the spatial attention map is as follows: ;in, Indicates learnable parameters, This represents a 2D skeleton keypoint diagram. This represents the basic attention map.

4. The motion quality evaluation method based on foreground enhancement according to claim 1, characterized in that, The expression for the spatial action encoding is as follows: ;in, This represents the third and fourth layers of the I3D network. Representing a spatial attention map, It represents the basic spatial characteristics.

5. The motion quality evaluation method based on foreground enhancement according to claim 1, characterized in that, The temporal encoding is obtained by fusing temporal segmentation representation and implicit topological structure using a cross-attention mechanism.

6. The motion quality evaluation method based on foreground enhancement according to claim 5, characterized in that, The expression for the cross-attention mechanism is as follows: ;in, Represents spatial action encoding, This represents a 2D skeleton keypoint diagram. ( ) represents a DETR-style time decoder. () represents the InfoGCN algorithm.

7. The motion quality evaluation method based on foreground enhancement according to claim 1, characterized in that, The prediction module is constructed using a contrastive regression calculator, which calculates the score difference between the time-coded example video and the time-coded video to be evaluated to obtain the corresponding predicted score as the motion quality evaluation result of the video to be evaluated compared to the example video.

8. A motion quality evaluation device, characterized in that, The steps for performing the motion quality evaluation method based on foreground enhancement as described in any one of claims 1-7.

Images