Method for evaluating motion quality based on mass fraction decoupling

Abstract

The application discloses a motion quality evaluation method based on quality score decoupling, collects a plurality of video samples according to actual needs, carries out sampling and pretreatment to obtain video clips and carries out label annotation, constructs a motion quality evaluation model comprising a dynamic flow feature encoder, a static flow feature encoder, a quality score decoupling module and a pair ordering module, trains the motion quality evaluation model by using the labeled video samples, samples and pretreats a video pair requiring motion quality evaluation by using the same method, obtains video clips and inputs the video clips into the motion quality evaluation model, and obtains an evaluation result. The application decouples the segment features of the video into high / low quality score features, carries out motion quality evaluation and prediction ordering on the pair of videos, and improves the motion quality evaluation accuracy.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of action quality evaluation, and more specifically relates to an action quality evaluation method based on quality score decoupling. BACKGROUND

[0002] The action quality evaluation task aims to design a system capable of automatically and objectively evaluating some specific actions completed in an input video. The action quality evaluation task has a wide range of application scenarios, such as surgical skill rating and medical rehabilitation training in medical care, athlete posture correction training systems and artificial intelligence referee systems in sports, operation compliance analysis and dangerous behavior monitoring in industrial production, etc.

[0003] With the development of computer technology, under the support of massive data and high-performance computing devices, more and more video understanding and analysis methods have been proposed, and the action quality evaluation task has also begun to attract widespread attention. At present, the long video action quality evaluation in the form of pairwise ranking generally uses the following steps: video segment feature extraction, video feature aggregation, score prediction and pairwise ranking result prediction.

[0004] Due to the limitation of convolutional neural networks in processing video length, the action quality evaluation method of long video generally adopts the following steps: first, uniformly sampling the video into multiple segments, and then aggregating the features of these segments based on the assumption that the person performs the same level in the whole video, so the action quality score of any part of the video is the same. However, in actual sports, this assumption does not necessarily hold, and the distribution of quality scores in long videos is generally uneven, which can lead to large errors in action quality evaluation and needs to be further improved. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide an action quality evaluation method based on quality score decoupling, which decouples the segment features of the video into high / low quality score features to predict and rank the results of action quality evaluation of paired videos, thereby improving the accuracy of action quality evaluation.

[0006] In order to achieve the above-mentioned purpose of the application, the action quality evaluation method based on quality score decoupling comprises the following methods:

[0007] S1: According to actual needs, a plurality of video samples are collected, each video sample is uniformly sampled to obtain N video segments with K frame numbers, and then all video frames are preprocessed using a preset method to obtain processed video segments; each video sample is labeled: let the two videos in the video pair be v i 、v j , if the video v i is better than the video vj the action quality of the video v i is better than that of the video v j , the label is set to 1, and if the action quality of the video v

[0008] S2: constructing an action quality evaluation model, including a dynamic stream feature encoder, a static stream feature encoder, a quality score decoupling module, a cascading module and a pairwise ranking module, wherein:

[0009] The dynamic stream feature encoder is used to extract dynamic stream features from the two videos v i and v j in the input video pair respectively, and the obtained dynamic stream features F Di and F Dj are sent to the quality score decoupling module. The specific method of dynamic stream feature extraction is: feature extraction is performed on N video segments of a video respectively, a feature vector with a dimension of M is extracted for each video segment, and the feature vectors of the N video segments are used as row vectors to form a feature matrix with a size of NXM, which is the dynamic stream feature of the video.

[0010] The static stream feature encoder is used to extract static stream features from the two videos v i and v j in the input video pair respectively, and the obtained static stream features F Si and F Sj are sent to the quality score decoupling module. The specific method of static stream feature extraction is: feature extraction is performed on N video segments of a video respectively, a feature vector with a dimension of M is extracted for each video segment, and the feature vectors of the N video segments are used as row vectors to form a feature matrix with a size of NXM, which is the dynamic stream feature of the video.

[0011] The quality score decoupling module is used to decouple the dynamic stream features and the static stream features to obtain high-quality score features and low-quality score features, and send them to the cascading module together with the average quality score features obtained by average pooling. The quality score decoupling module includes a dynamic stream feature average pooling module, a dynamic quality score decoder, a static stream feature average pooling module and a static quality score decoder, wherein:

[0012] The dynamic stream feature average pooling module is used to average the row vectors of the dynamic stream features F Di and F Dj to obtain average feature vectors with a dimension of M as dynamic stream average quality score features F DUi and F DUj .

[0013] The dynamic quality score decoder is used to respectively decode the dynamic stream features FDi Dj decoded, respectively, to obtain a dynamic score feature matrix of size 2xM, and a first row vector in the corresponding dynamic score feature matrix is taken as a dynamic flow high-quality score feature F DPi DPj decoded, respectively, to obtain a dynamic score feature matrix of size 2xM, and a first row vector in the corresponding dynamic score feature matrix is taken as a dynamic flow high-quality score feature F DCi DCj .

[0014] The static flow feature average pooling module is configured to perform row vector averaging on the static flow features F Si Sj , respectively, to obtain an average feature vector of dimension M as a static flow average quality score feature F SUi SUj .

[0015] The static quality score decoder is configured to decode the static flow features F Si Sj , respectively, to obtain a static score feature matrix of size 2xM, and a first row vector in the corresponding static score feature matrix is taken as a static flow high-quality score feature F SPi SPj , and a second row vector is taken as a dynamic flow low-quality score feature F SCi SCj .

[0016] The dynamic quality score decoder and the static quality score decoder are implemented based on a Transformer decoder, and the dynamic quality score decoder / static quality score decoder includes G layers of networks, each layer of network including an attention module, a prediction feedforward network, and a feature fusion module, wherein:

[0017] The attention module is configured to process received features using an attention mechanism to obtain a feature matrix of size 2xM and send the feature matrix to the prediction feedforward network and the feature fusion module, wherein the query q g , the key k g , and the value v g in the attention mechanism of the attention module in the gth layer of network are calculated in the following manner:

[0018] q g = WA qg F g

[0019] k g = WA kg F

[0020] v g = WA vg F

[0021] ​​​​​​​​wherein g = 1, 2, …, G, when g = 1, P1 is an M-dimensional vector with a size of 2 x M, representing a high-quality feature prototype of the first layer that can be learned, C1 is an M-dimensional vector, representing a low-quality feature prototype of the first layer that can be learned, when g = 2, 3, …, G, P g is an M-dimensional vector, representing a high-quality feature prototype of the gth layer that can be learned, C g is an M-dimensional vector, representing a low-quality feature prototype of the gth layer that can be learned, O g-1 represents a 2 x M feature matrix output by the previous layer network, and Self-attention() represents a self-attention mechanism; F has a dimension of N x M, representing a dynamic stream feature or a static stream feature of the input video; W qg , W kg and W vg respectively represent three linear projection matrices that can be learned in the gth layer;

[0022] The prediction feedforward network is used to process the received features to obtain a feature matrix with a size of 2 x M and output to the feature fusion module;

[0023] The feature fusion module is used to add the feature matrix output by the attention module and the feature matrix output by the prediction feedforward network to obtain a fusion feature matrix. The feature fusion module of the first G-1 layer outputs the fusion feature matrix to the attention module of the next layer, and the feature fusion module of the Gth layer outputs the fusion feature matrix as a dynamic score feature matrix or a static score feature matrix. The first row vector is a dynamic stream / static stream high-quality score feature, and the second row vector is a dynamic stream / static stream low-quality score feature.

[0024] The cascading module is used to cascade the dynamic stream quality score feature and the static stream score feature of the same video, and send the cascaded score feature to the pairwise ranking module. The specific method of feature cascading is as follows:

[0025] For a video v i , the dynamic stream average quality score feature F DUi and the static stream average quality score feature F SUi are cascaded to obtain the average quality score feature F Ui , the dynamic stream high-quality score feature F DPi and the static stream high-quality score feature F SPi are cascaded to obtain the high-quality score feature F Pi , the dynamic stream low-quality score feature F DCi and the static stream low-quality score feature F SCi are cascaded to obtain the low-quality score feature F Ci .

[0026] For video v j The dynamic flow average mass fraction characteristic F DUj and static flow average mass fraction characteristic F SUj The average mass fraction characteristic F is obtained by cascading. Uj The high-quality score feature F of the dynamic flow DPj and static flow high quality score feature F SPj Cascading yields high-quality fractional features F Pj The low quality fraction feature F of the dynamic flow DCj and the low quality fraction characteristic F of static flow SCj Cascading yields low-quality-score features F Cj ;

[0027] The pairwise ranking module is used to rank the two received videos based on their score features, thus determining the winning video. The pairwise ranking module includes a quality score predictor, an action quality score summation module, and a pairwise ranking predictor, wherein:

[0028] The quality score predictor predicts three motion quality scores based on three input score features and sends them to the motion quality score summing module, where video v i Average mass fraction characteristic F Ui The predicted motion quality score is Sr(F) Ui High-quality score feature F Pi The predicted motion quality score is Sr(F) Pi Low quality score feature F Ci The predicted motion quality score is Sr(F) Ci ); Video v j Average mass fraction characteristic F Uj The predicted motion quality score is Sr(F) Uj High-quality score feature F Pj The predicted motion quality score is Sr(F) Pj Low quality score feature F Cj The predicted motion quality score is Sr(F) Cj );

[0029] The motion quality score summation module is used to sum the quality scores of three motions in the same video to obtain video v. i Action quality score S i =Sr(F Ui )+Sr(F Pi )+Sr(F Ci ) and video v j Action quality score S j =Sr(FUj )+Sr(F Pj )+Sr(F Cj ), and then the action quality scores of the two videos are sent to a pairwise ranking predictor;

[0030] The pairwise ranking predictor is used to compare the action quality scores of the videos v i and v j , and output a video winning result, specifically as follows: if S i >S j , then the video v i wins, and the prediction evaluation result is 1; if S i <S j , then the video v j wins, and the prediction evaluation result is -1.

[0031] S3: using the video segments of the two videos in the video pair in step S1 as input and the label as expected output, training the action quality evaluation model to obtain a trained action quality evaluation model;

[0032] S4: for a video pair that needs to be evaluated for action quality, each video is sampled and preprocessed according to the same method in step S1 to obtain video segments and input the video segments into the action quality evaluation model to obtain an evaluation result.

[0033] The present application is based on an action quality evaluation method based on quality score decoupling, a plurality of video samples are collected according to actual needs, video segments are obtained by sampling and preprocessing and are labeled, an action quality evaluation model including a dynamic stream feature encoder, a static stream feature encoder, a quality score decoupling module and a pairwise ranking module is constructed, the action quality evaluation model is trained using the labeled video samples, for a video pair that needs to be evaluated for action quality, the same method is used for sampling and preprocessing to obtain video segments and input the video segments into the action quality evaluation model to obtain an evaluation result.

[0034] The present application can evaluate the action quality of a pair of videos input by a user, use an action quality score decoupling method to decouple the segment features of a video into high / low quality score features, and use an average pooling feature as an average quality score feature, so that the problem of uneven distribution of quality scores in long video action quality evaluation in a pairwise ranking manner can be solved by score alignment, and the accuracy of action quality evaluation is improved. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 is a flowchart of a specific embodiment of the action quality evaluation method based on quality score decoupling of the present application;

[0036] Figure 2is a structural diagram of the action quality evaluation model in the present application;

[0037] Figure 3 is a structural diagram of the dynamic flow feature encoder in the present embodiment;

[0038] Figure 4 is a structural diagram of the static flow feature encoder in the present embodiment;

[0039] Figure 5 is a structural diagram of the dynamic quality score decoder / static quality score decoder in the present application. DETAILED DESCRIPTION

[0040] The specific embodiments of the present application will be described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present application. It should be particularly noted that in the following description, when detailed description of known functions and designs may dilute the main content of the present application, these descriptions will be omitted here.

[0041] EMBODIMENT

[0042] Figure 1 is a specific implementation flowchart of the action quality evaluation method based on quality score decoupling in the present application. As shown in Figure 1 , the specific steps of the action quality evaluation method based on quality score decoupling in the present application include:

[0043] S101: Training sample preprocessing:

[0044] According to actual needs, a plurality of video sample pairs and the winning video sample in each video sample pair are collected, N video segments with K frame numbers are obtained by uniformly sampling each video sample, and then all video frames are preprocessed by a preset method to obtain processed video segments. Labeling is performed on each video sample pair: let the two videos in the video pair be v i , v j , if the action quality of the video v i is better than that of the video v j , then the label is 1, and if the action quality of the video v i is worse than that of the video v j , then the label is -1.

[0045] The preprocessing method of the video frame in the present embodiment is to normalize the video frame to a preset size and then perform center cropping. In the present embodiment, each video in the video pair is uniformly sampled into 256 video segments of 16 frames, and all frames are scaled to a height of 256 and center cropped to 224x224.

[0046] S102: Constructing an action quality evaluation model:

[0047] In order to realize the action quality evaluation, the action quality evaluation model is provided based on quality score decoupling through research. Figure 2 is a structural diagram of the action quality evaluation model in the application. The action quality evaluation model in the application comprises a dynamic stream feature encoder, a static stream feature encoder, a quality score decoupling module, a cascading module and a pairwise ranking module, and each module will be described in detail.

[0048] The dynamic stream feature encoder is used for extracting dynamic stream features of two videos v i , v j in the input video pair respectively, and sending the obtained dynamic stream features F Di , F Dj to the quality score decoupling module. i , v j in the input video pair respectively, and sending the obtained dynamic stream features F Di , F Dj to the quality score decoupling module. The specific method of dynamic stream feature extraction is as follows: feature extraction is performed on N video segments of a video respectively, a feature vector with a dimension of M is extracted for each video segment, and the feature vectors of the N video segments are taken as row vectors to form a feature matrix with a size of NXM, which is taken as the dynamic stream feature of the video.

[0049] In this embodiment, the dynamic stream feature encoder adopts an I3D (Inflated 3D ConvNets) network. The I3D network is a three-dimensional convolutional neural network used for extracting features of video segments. The I3D directly expands the size of a 2D convolution kernel with a size of n*n to a 3D convolution kernel with a size of n*n*n by increasing the time dimension of a pre-trained 2D CNN, so that the pre-trained 2D CNN can be well used to complete the feature extraction of the video segments. Figure 3 is a structural diagram of the dynamic stream feature encoder in this embodiment. As shown in Figure 3 , the video dimension input by the dynamic stream feature encoder in this embodiment is 16*224*224*3, Conv3d_n*n represents a three-dimensional convolution with a convolution kernel of n*n*n, MaxPool3d and AvgPool3d represent three-dimensional maximum pooling and average pooling, Inception*n represents an Inception network structure, a plurality of convolution kernels with different kernel sizes and pooling operations are cascaded as an output in the I3D network, and n represents consecutive n Inception network structures. Finally, a feature matrix with a size of 256*1024 is obtained.

[0050] The static stream feature encoder is used for extracting static stream features of two videos v i , v j in the input video pair respectively, and sending the obtained static stream features F Si , F SjThe quality score decoupling module is used for decoupling the dynamic flow feature and the static flow feature to obtain a high-quality score feature and a low-quality score feature, and sending the average quality score feature obtained by average pooling to the cascading module together. The quality score decoupling module comprises a dynamic flow feature average pooling module, a dynamic quality score decoder, a static flow feature average pooling module and a static quality score decoder, wherein: The specific method of static flow feature extraction is as follows: for N video clips of a video, a frame of image is extracted in each video clip for feature extraction, a feature vector with a dimension of M is extracted from each frame of image, and the feature vectors of N frames of image are taken as row vectors to form a feature matrix with a size of N*M, which is taken as the static flow feature of the video.

[0051] In this embodiment, the static flow feature encoder adopts a ResNet50 network. The ResNet50 network is a ResNet (Deep Residual Network) model with a depth of 50, and is a commonly used image feature extraction model. Figure 4 is a structural diagram of the static flow feature encoder in this embodiment. As shown in Figure 4 , the input image dimension of the static flow feature encoder in this embodiment is 224*224*3, Conv_7*7_64 (s2) represents that the convolution kernel is 7*7, the output channel is 64, the step is 2, Relu refers to the activation function relu, BN (Batch Normalization) represents batch normalization, Res Block represents residual block, there are a total of four groups of Block, each group contains 3, 4, 6 and 3 residual blocks respectively, each residual block contains three convolution layers, FC represents a fully connected layer, Max_pooling and Avg_pooling represent maximum pooling and average pooling respectively.

[0052] The quality score decoupling module is used for decoupling the dynamic flow feature and the static flow feature to obtain a high-quality score feature and a low-quality score feature, and sending the average quality score feature obtained by average pooling to the cascading module together. The quality score decoupling module comprises a dynamic flow feature average pooling module, a dynamic quality score decoder, a static flow feature average pooling module and a static quality score decoder, wherein:

[0053] The dynamic flow feature average pooling module is used for performing row vector average on the dynamic flow features F Di and F Dj respectively to obtain average feature vectors with a dimension of M as the dynamic flow average quality score features F DUi and F DUj .

[0054] The dynamic quality score decoder is used for decoding the dynamic flow features F Di and F Dj respectively to obtain dynamic score feature matrices with a size of 2*M, taking the first row vectors of the corresponding dynamic score feature matrices as the dynamic flow high-quality score features F DPi and F DPj , and taking the second row vectors as the dynamic flow low-quality score features FDCi , F DCj .

[0055] The static flow feature average pooling module is used for respectively performing row vector average on the static flow features F Si , F Sj , and obtaining an average feature vector with a dimension of M as a dynamic flow average quality score feature F SUi , F SUj .

[0056] The static quality score decoder is used for respectively decoding the static flow features F Si , F Sj , and respectively obtaining a static score feature matrix with a size of 2xM, taking the first row vector in the corresponding static score feature matrix as a static flow high-quality score feature F SPi , F SPj , and taking the second row vector as a static flow low-quality score feature F SCi , F SCj .

[0057] The dynamic quality score decoder and the static quality score decoder in the application are implemented based on a Transformer decoder. Figure 5 is a structural diagram of the dynamic quality score decoder / static quality score decoder in the application. The dynamic quality score decoder and the static quality score decoder use the same structure, but do not share parameters. As shown in Figure 5 , the dynamic quality score decoder / static quality score decoder in the application contains G layers of networks, each layer of network containing an attention module, a prediction feedforward network and a feature fusion module, wherein:

[0058] The attention module is used for processing the received features by adopting an attention mechanism, obtaining a feature matrix with a size of 2xM and sending it to the prediction feedforward network and the feature fusion module, wherein the query q g , the key k g and the value v g in the attention mechanism of the attention module in the gth layer of network are calculated in the following manner:

[0059] q g = W qg A g

[0060] k g = W kg F

[0061] v g = W vg F

[0062] , wherein g = 1, 2, …, G, when g = 1, is a 2xM matrix, P1is an M-dimensional vector, representing the high-quality feature prototype of the first learnable layer, C1is an M-dimensional vector, representing the low-quality feature prototype of the first learnable layer, when g = 2, 3, …, G, is a 2xM matrix, P g is an M-dimensional vector, representing the high-quality feature prototype of the gth learnable layer, C g is an M-dimensional vector, representing the low-quality feature prototype of the gth learnable layer, O g-1 is a 2xM feature matrix output by the previous layer network, and Self-attention() represents the self-attention mechanism. That is, when g = 2, 3, …, G, the query in the attention mechanism is a 2xM matrix obtained by adding the matrix composed of the high-quality feature prototype and the low-quality feature prototype to the feature matrix output by the previous layer network and then processing it through the self-attention mechanism. The dimension of F is N x M, representing the dynamic flow feature or static flow feature of the input video v i . qg , W kg , and W vg respectively represent three linear projection matrices learned by the gth layer.

[0063] According to the above description, the query q g is obtained based on the high-quality feature prototype and the low-quality feature prototype through the linear projection layer, the key k g , and the value v g are obtained from the dynamic feature flow or the static feature flow through different linear projection layers.

[0064] The attention mechanism is a commonly used deep learning learning technique, and its specific calculation process is not described here.

[0065] The prediction feedforward network is used to process the received features to obtain a 2xM feature matrix and output it to the feature fusion module.

[0066] The feature fusion module is used to add the feature matrix output by the attention module and the feature matrix output by the prediction feedforward network to obtain a fusion feature matrix. The feature fusion module of the first G-1 layer outputs the fusion feature matrix to the attention module of the next layer, and the feature fusion module of the Gth layer outputs the fusion feature matrix as a dynamic score feature matrix or a static score feature matrix. The first row vector is the dynamic flow / static flow high-quality score feature, and the second row vector is the dynamic flow / static flow low-quality score feature.

[0067] The cascading module is used to cascade the dynamic flow quality score feature and the static flow feature score feature of the same video, and send the cascaded score feature to the pairwise ranking module. The specific method of feature cascading is:

[0068] For video v i The dynamic flow average mass fraction characteristic F DUi and static flow average mass fraction characteristic F SUi The average mass fraction characteristic F is obtained by cascading. Ui The high-quality score feature F of the dynamic flow DPi and static flow high quality score feature F SPi Cascading yields high-quality fractional features F Pi The low quality fraction feature F of the dynamic flow DCi and the low quality fraction characteristic F of static flow SCi Cascading yields low-quality-score features F Ci .

[0069] For video v j The dynamic flow average mass fraction characteristic F DUj and static flow average mass fraction characteristic F SUj The average mass fraction characteristic F is obtained by cascading. Uj The high-quality score feature F of the dynamic flow DPj and static flow high quality score feature F SPj Cascading yields high-quality fractional features F Pj The low quality fraction feature F of the dynamic flow DCj and the low quality fraction characteristic F of static flow SCj Cascading yields low-quality-score features F Cj .

[0070] The pairwise ranking module is used to rank two received videos based on their score features, thus determining the winning video. The pairwise ranking module includes a quality score predictor, an action quality score summation module, and a pairwise ranking predictor, wherein:

[0071] The quality score predictor predicts three motion quality scores based on three input score features and sends them to the motion quality score summing module, where video v i Average mass fraction characteristic F Ui The predicted motion quality score is Sr(F) Ui High-quality score feature F Pi The predicted motion quality score is Sr(F) Pi Low quality score feature F Ci The predicted motion quality score is Sr(F) Ci ); Video v j Average mass fraction characteristic F Uj The predicted motion quality score is Sr(F) Uj High-quality score feature F Pj The predicted motion quality score is Sr(F)Pj ), low-quality-score feature F Cj The predicted action quality score is Sr(F Cj ).

[0072] The quality score predictor in this embodiment includes four layers of fully connected layers in cascade, wherein the first three layers of fully connected layers adopt ReLU activation functions, and the fourth layer of fully connected layers adopts a Sigmoid activation function. The dimension of the input feature is 2048, which is sequentially reduced to 512, 256, and 128 through three FC layers and ReLU activation functions, and finally an action quality score is output through a fully connected layer and a Sigmoid activation function. The Sigmoid activation function is used to normalize the predicted score to the range of [0, 1].

[0073] The action quality score summation module is used to sum the three action quality scores of the same video, to obtain the action quality score S i of the video v i = Sr(F Ui )+ Sr(F Pi )+ Sr(F Ci ) and the action quality score S j of the video v j = Sr(F Uj )+ Sr(F Pj )+ Sr(F Cj ), and then the action quality scores of the two videos are sent to the pairwise ranking predictor.

[0074] The pairwise ranking predictor is used to compare the action quality scores of the video v i and the video v j , and output the video evaluation result. The specific method is: if S i > S j , the video v i wins, and the predicted evaluation result is 1; if S i < S j , the video v j wins, and the predicted evaluation result is -1.

[0075] S103: Training the action quality evaluation model:

[0076] The video segments of the two video pairs in the video pair in step S101 are used as input, and the label is used as the expected output. The action quality evaluation model is trained to obtain the trained action quality evaluation model.

[0077] In order to improve the training effect, the action quality evaluation model in this embodiment adopts step-by-step training. The specific method is:

[0078] First, the dynamic stream feature encoder and the static stream feature encoder are pre-trained respectively using the data set. In this embodiment, the dynamic stream feature encoder uses the I3D network, and the Kinetics data set is used for training. The static stream feature encoder uses the ResNet50 network, and the ImageNet data set is used for training.

[0079] Then, the parameters of the dynamic feature encoder and the static stream feature encoder are fixed, and the quality score decoupling module and the pair ranking module are jointly trained.

[0080] When the quality score decoupling module and the pair ranking module are jointly trained, in order to improve the training effect, the loss function is optimized and set. In this embodiment, the calculation formula of the loss function LOSS is as follows:

[0081] LOSS=L rank +L dist +L aware +λL div

[0082] Wherein, λ represents a preset weight parameter, L rank represents the pair ranking loss, and the calculation formula is as follows:

[0083]

[0084]

[0085]

[0086]

[0087] Wherein, represents the loss of the high-quality score feature, represents the loss of the low-quality score feature, represents the loss of the average quality score feature, and m1 represents a preset hyperparameter for constraining the difference.

[0088] L dist represents the difference loss, and the calculation formula is as follows:

[0089]

[0090]

[0091]

[0092] wherein m2 represents a preset hyperparameter for constraining the predicted action quality score difference. The difference loss encourages the quality score decoupling module to obtain more diverse high / low quality score features by constraining the predicted action quality score difference of the features obtained by the quality score decoupling module to be greater than the predicted action quality score difference of the average quality score feature. Specifically, the action quality score difference values of the two video high / low quality score features predicted by m2 are all greater than the action quality score difference value of the average quality score feature.

[0093] L aware represents the ranking-aware loss, and the calculation formula is as follows:

[0094] L aware = (0, m3- (S r (F Pi )-S r (F Cj )) + (S r (F Ui )-S r (F Uj ))

[0095] wherein m3 represents a preset hyperparameter for constraining the high / low action quality score difference. The pair-wise ranking loss and the difference loss cannot constrain the features decoupled to represent the high quality score features and the low quality score features, respectively. The ranking-aware loss constrains the predicted score of the high quality score feature of the video v i to be much greater than the predicted score of the low quality score feature of the video v j , thereby constraining the two features decoupled to represent the high quality score features and the low quality score features, respectively.

[0096] L div represents the diversity loss, and the calculation method is as follows:

[0097] The corresponding dynamic attention matrix A i and the static attention matrix A Di of the video v Si in the last layer of the network are extracted from the dynamic quality score decoder and the static quality score decoder, respectively. j The corresponding dynamic attention matrix A Dj and the static attention matrix A Sj of the video v div in the last layer of the network are extracted from the dynamic quality score decoder and the static quality score decoder, respectively.

[0098]

[0099]

[0100]

[0101] wherein the superscript T denotes the transpose, I denotes the identity matrix, and || || F denotes the Frobenius norm. F denotes the Frobenius norm.

[0102] Unlike the previous three loss functions, the diversity loss directly acts on the quality score decoupling module, and constrains the high / low quality score features to focus on different video clips respectively.

[0103] The hyperparameter settings for jointly training the quality score decoupling module and the pair-wise ranking module in the embodiment are as follows: m1=1, m2=0.05, m3=0.15, and λ=1. The batch data size is set to 16, and 500 epochs are trained. The adaptive matrix estimation Adam optimizer is used, and the settings are as follows: the weight decay is 0.0005, and the learning rate is 0.0001.

[0104] S104: Action quality evaluation:

[0105] For the video pairs that need to be evaluated for action quality, each video is sampled and preprocessed according to the same method in step S101, video clips are obtained, and the action quality evaluation model is input to obtain the evaluation result.

[0106] In order to better illustrate the technical effects of the present application, specific examples are used to experimentally verify the present application. In this experimental verification, the daily life skill video dataset EPIC-Skills 2018 and BEST and the self-collected competitive sports video dataset Breaking dance action quality evaluation dataset are used as experimental datasets, and five kinds of comparative methods are selected for comparative experiments, and the correct evaluation rate is counted. The five kinds of comparative methods are as follows:

[0107] Doughty et al. algorithm, see the literature "Doughty H, Mayol-Cuevas W, Damen D. The pros and cons: Rank-aware temporal attention for skill determination in long videos [C]. Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019: 7862-7871."

[0108] I3D algorithm, see the document "Carreira J, Zisserman A. Quo vadis, action recognition? a new model and the kinetics dataset [C]. proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6299-6308."

[0109] Li et al. algorithm, see the document "Li Z, Huang Y, Cai M, et al. Manipulation-skill assessment from videos with spatial attention network [C]. Proceedings of the IEEE / CVF International Conference on Computer Vision Workshops, 2019: 0-0."

[0110] GDLT algorithm, see the document "Xu A, Zeng L A, Zheng W S. Likert scoring with grade decoupling for long-term action assessment [C]. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022: 3232-3241."

[0111] Pan et al. algorithm, see the document "Pan J H, Gao J, Zheng W S. Adaptive action assessment [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(12): 8779-8795."

[0112] Table 1 is a comparison table of pairwise ranking accuracy of the present application and the comparative method on the daily life skill video dataset EPIC-Skills 2018.

[0113] Method Chopstick-Using Surgery Drawing Dough-Rolling Avg.ACC Li et al. 85.5 73.1 85.3 82.7 81.6 Doughty et al. - - - - 80.3 Pan et al. 87.7 71.9 88.1 88.5 84.1 The invention 84.7 74.0 88.7 87.8 83.8

[0114] Table 1

[0115] Table 2 is a pair-wise ranking accuracy comparison table of the present application and the comparative method on the daily life skill video dataset BEST.

[0116] Method eggs Braid_hair tie_tie origami apply_eyeliner Avg.ACC Doughty et al. 87.7 74.8 89.3 78.7 84.7 81.2 GDLT 83.4 72.2 75.3 78.0 74.8 76.7 Pan et al. 83.8 75.2 89.2 82.6 85.5 83.3 The invention 90.4 77.1 85.1 83.0 88 84.7

[0117] Table 2

[0118] Table 3 is a pair-wise ranking accuracy comparison table of the present application and the comparative method on the competitive sports dataset Breaking dance motion quality evaluation dataset.

[0119] Method ACC Doughty et al. 75.4 I3D 72.6 Li et al. 72.8 GDLT 78.7 The invention 80.9

[0120] Table 3

[0121] As shown in Tables 1 to 3, the pair-wise ranking accuracy of the present application is better than that of the prior art method, because the network structure combining dynamic and static in the present application can extract more abundant features, the quality score decoupling module decouples the high / low quality score features through the part related to skill level in the long video, and the features of the three branches use the common score predictor to improve the performance of the score predictor, thereby obtaining better pair-wise ranking prediction results.

[0122] Although the above describes the illustrative specific embodiments of the present application in order to facilitate the understanding of the present application by those skilled in the art, it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

Claims

1. A motion quality assessment method based on quality score decoupling, characterized in that, Includes the following steps: S1: Collect several pairs of video samples according to actual needs. For each video sample, perform uniform sampling to obtain N video segments with K frames. Then, preprocess all video frames using a preset method to obtain processed video segments. Label each video sample pair: Let v be the two videos in the video pair. i v j If video v i Compared to video v j If the motion quality is better, then the label is set to 1; if the video v i Compared to video v j If the quality of the action is worse, then the label is -1; S2: Construct an action quality assessment model, including a dynamic flow feature encoder, a static flow feature encoder, a quality score decoupling module, a cascade module, and a pairwise sorting module, wherein: Dynamic stream feature encoders are used to process two videos v in an input video pair. i v j Extract dynamic flow features separately, and then extract the dynamic flow features F. Di F Dj The data is sent to the quality score decoupling module. The specific method for dynamic flow feature extraction is as follows: feature extraction is performed on N video segments of the video. Each video segment extracts a feature vector of dimension M. The feature vectors of the N video segments are used as row vectors to form a feature matrix of size N×M. This feature matrix is ​​used as the dynamic flow feature of the video. A static stream feature encoder is used to process two videos v in an input video pair. i v j Extract the static flow features separately, and then extract the static flow features F. Si F Sj Send to the quality score decoupling module; The specific method for static stream feature extraction is as follows: For N video segments of a video, extract one frame of image from each video segment for feature extraction. Each frame of image extracts a feature vector of dimension M. Use the feature vectors of N frames of images as row vectors to form a feature matrix of size N×M. This feature matrix is ​​used as the static stream feature of the video. The quality score decoupling module decouples dynamic flow features and static flow features to obtain high-quality and low-quality-score features, respectively. These features, along with the average quality score feature obtained through average pooling, are then sent to the cascaded module. The quality score decoupling module includes a dynamic flow feature average pooling module, a dynamic quality score decoder, a static flow feature average pooling module, and a static quality score decoder, wherein: The dynamic flow feature average pooling module is used to process the dynamic flow feature F respectively. Di F Dj The average of the row vectors is used to obtain the average feature vector of dimension M, which is used as the dynamic flow average quality score feature F. DUi F DUj ; The dynamic quality score decoder is used to analyze the dynamic flow features F respectively. Di F Dj Decoding yields dynamic score feature matrices of size 2×M. The first row vector in the corresponding dynamic score feature matrix is ​​taken as the high-quality score feature F of the dynamic stream. DPi F DPj The second row vector is used as the low-quality score feature F of the dynamic flow. DCi F DCj ; The static flow feature average pooling module is used to process the static flow feature F separately. Si F Sj The average of the row vectors is used to obtain the average feature vector of dimension M, which is used as the static flow average quality score feature F. SUi F SUj ; The static quality score decoder is used to analyze the static stream features F respectively. Si F Sj Decoding yields static score feature matrices of size 2×M. The first row vector in the corresponding static score feature matrix is ​​taken as the high-quality score feature F of the static stream. SPi F SPj The second row vector is used as the low-quality-score feature F of the static flow. SCi F SCj ; The dynamic quality score decoder and static quality score decoder are implemented based on the Transformer decoder. Each dynamic / static quality score decoder consists of a G-layer network, with each layer containing an attention module, a prediction feedforward network, and a feature fusion module. The attention module processes the received features using an attention mechanism, obtaining a 2×M feature matrix, which is then sent to the prediction feedforward network and the feature fusion module. The attention mechanism in the g-th layer of the network queries q. g , key k g Sum of values ​​v g The calculation is performed as follows: q g =W qg A g k g =W kg F v g =W vg F Where g = 1, 2, ..., G, when g = 1, The vector is 2×M in size, where P1 is an M-dimensional vector representing the high-quality feature prototypes of the first learnable layer, and C1 is an M-dimensional vector representing the low-quality feature prototypes of the first learnable layer. When g = 2, 3, ..., G, Size is 2×M, P g Let C be an M-dimensional vector representing a high-quality feature prototype of the learnable g-th layer. g Let O be an M-dimensional vector, representing the low-quality feature prototype of the learnable g-th layer. g-1 Let F represent the 2×M feature matrix output by the previous layer, and Self-attention() represent the self-attention mechanism; F has a dimension of N×M, representing the dynamic or static flow features of the input video; W qg W kg and W vg These represent the three linear projection matrices that can be learned at the g-th layer; The prediction feedforward network is used to process the received features, obtain a feature matrix of size 2×M, and output it to the feature fusion module; The feature fusion module is used to add the feature matrix output by the attention module and the feature matrix output by the prediction feedforward network to obtain the fused feature matrix. The feature fusion module of the first G-1 layers outputs the fused feature matrix to the attention module of the next layer. The feature fusion module of the Gth layer outputs the fused feature matrix as a dynamic score feature matrix or a static score feature matrix. The first row vector is used as the high-quality score feature of dynamic flow / static flow, and the second row vector is used as the low-quality score feature of dynamic flow / static flow. The concatenation module is used to concatenate the dynamic stream quality score features and static stream quality score features of the same video, and then send the concatenated score features to the pairwise sorting module. The specific method of feature concatenation is as follows: For video v i The dynamic flow average mass fraction characteristic F DUi and static flow average mass fraction characteristic F SUi The average mass fraction characteristic F is obtained by cascading. Ui The high-quality score feature F of the dynamic flow DPi and static flow high quality score feature F SPi Cascading yields high-quality fractional features F Pi The low quality fraction feature F of the dynamic flow DCi and the low quality fraction characteristic F of static flow SCi Cascading yields low-quality-score features F Ci ; For video v j The dynamic flow average mass fraction characteristic F DUj and static flow average mass fraction characteristic F SUj The average mass fraction characteristic F is obtained by cascading. Uj The high-quality score feature F of the dynamic flow DPj and static flow high quality score feature F SPj Cascading yields high-quality fractional features F Pj The low quality fraction feature F of the dynamic flow DCj and the low quality fraction characteristic F of static flow SCj Cascading yields low-quality-score features F Cj ; The pairwise ranking module is used to rank the two received videos based on their score features, thus determining the winning video. The pairwise ranking module includes a quality score predictor, an action quality score summation module, and a pairwise ranking predictor, wherein: The quality score predictor predicts three motion quality scores based on three input score features and sends them to the motion quality score summing module, where video v i Average mass fraction characteristic F Ui The predicted motion quality score is Sr(F) Ui High-quality score feature F Pi The predicted motion quality score is Sr(F) Pi Low quality score feature F Ci The predicted motion quality score is Sr(F) Ci ); Video v j Average mass fraction characteristic F Uj The predicted motion quality score is Sr(F) Uj High-quality score feature F Pj The predicted motion quality score is Sr(F) Pj Low quality score feature F Cj The predicted motion quality score is Sr(F) Cj ); The motion quality score summation module is used to sum the quality scores of three motions in the same video to obtain video v. i Action quality score S i =Sr(F Ui )+Sr(F Pi )+Sr(F Ci ) and video v j Action quality score S j =Sr(F Uj )+Sr(F Pj )+Sr(F Cj Then, the motion quality scores of the two videos are sent to the pairwise ranking predictor; Pairwise sorting predictors are used for video v i and video v j The motion quality scores are compared, and the winning video is output. The specific method is as follows: if S i >S j Then video v i If S wins, the predicted evaluation result is 1; i <S j Then video v j Winner, predicted assessment result is -1; S3: Using video clips from two video pairs in step S1 as input and labels as expected output, train the motion quality assessment model to obtain a trained motion quality assessment model. S4: For video pairs that require motion quality assessment, sample and preprocess each video using the same method as in step S1 to obtain video segments, which are then input into the motion quality assessment model to obtain the assessment results.

2. The motion quality assessment method according to claim 1, characterized in that, The dynamic flow feature encoder uses an I3D network.

3. The motion quality assessment method according to claim 1, characterized in that, The static stream feature encoder uses a ResNet50 network.

4. The motion quality assessment method according to claim 1, characterized in that, The quality score predictor comprises four cascaded fully connected layers, wherein the first three fully connected layers use the ReLU activation function and the fourth fully connected layer uses the Sigmoid activation function.

5. The motion quality assessment method according to claim 1, characterized in that, The training of the motion quality assessment model in step S3 adopts step-by-step training. The specific method is as follows: first, the dynamic flow feature encoder and the static flow feature encoder are pre-trained using the dataset respectively. Then, the parameters of the dynamic feature encoder and the static flow feature encoder are fixed, and the quality score decoupling module and the pairwise sorting module are jointly trained.

6. The motion quality assessment method according to claim 5, characterized in that, The calculation formula for the loss function LOSS used when the quality score decoupling module and the pairwise ranking module are jointly trained is as follows: LOSS=L rank +L dist +L aware +λL div Where λ represents the preset weight parameter, L rank The loss for pairwise sorting is expressed by the following formula: in, The loss represents the high-quality score features. The loss represents the low-quality-score features. The loss represents the average quality fraction feature, and m1 represents the preset hyperparameter used to constrain the difference. L dist The difference loss is expressed by the following formula: Where m2 represents a preset hyperparameter used to constrain the difference in predicted action quality scores; L aware The order-aware loss is represented by the following formula: L aware =(0,m3-(S r (F Pi )-S r (F Cj ))+(S r (F Ui )-S r (F Uj ))) Where m3 represents a preset hyperparameter used to constrain the difference in quality scores between high and low motions; L div The diversity loss is represented and calculated as follows: Extract the video v from the last layer of the network from the dynamic quality score decoder and the static quality score decoder respectively. i The corresponding dynamic attention matrix A Di and static attention matrix A Si And the video v in the last layer of the network j The corresponding dynamic attention matrix A Dj and static attention matrix A Sj Then, the diversity loss L is calculated using the following formula. div : Where the superscript T denotes transpose, I denotes identity matrix, and || || F This indicates the search for the Frobenius norm.

Images