Uncertain action quality assessment method based on video interaction enhancement

Abstract

The invention discloses an uncertainty action quality evaluation method based on video interaction enhancement, which is based on a visual framework and comprises the following steps of: firstly, establishing a single-person movement video data set, preprocessing a video, extracting features, exploring information interaction among video clips by using a self-attention mechanism, and evaluating the motion quality of the single-person movement video data set; the method comprises the following steps of: selecting a plurality of fragments, changing the fragments into mutually related groups instead of independent groups, endowing adaptive weights to the fragments, simulating a real scoring scene in which the completeness of the actions is greatly influenced by important actions, and finally, learning predictable score probability by using an uncertainty perception score distribution learning network, according to the method, the most advanced performance is realized on the action quality evaluation data set.

Description

Technical Field

[0001] The present invention belongs to the field of video understanding, and more specifically, relates to an uncertainty action quality assessment method based on video interaction enhancement. Background Art

[0002] Motion quality assessment aims to evaluate the skill level of humans in specific scenarios. It is becoming one of the important topics in the field of computer vision. Its application range is very wide, including skill assessment in sports, sports rehabilitation, and daily life.

[0003] Almost all existing work currently treats AQA as a regression problem. Specifically, after dividing a video into segments, a feature extraction network is used to extract clip-level features from the video. These aggregated features are then fed into a linear regressor to predict a score based on these aggregated features. However, in the real world, AQA evaluates continuous movements. Not only does the degree of movement completion affect the judges' score, but the coherence between movement transitions also affects the final result. For example, in diving, athletes complete the movements from takeoff to entry with a high degree of completion. However, the smoothness of the transition between these two movements varies, resulting in different final scores.

[0004] In addition, for the subsequent processing of the obtained clip features, there are currently two main processing methods: the first is to simply average the clips to aggregate the clips to ensure that the information of each video clip is equally considered. The second method uses some fine-grained annotated datasets to divide the action into sub-stages. For example, diving actions are divided into four stages: approach, take-off, air, and entry into the water, to obtain the relative score of each clip, and the total score is obtained by weighted addition. We believe that the quality of the action is not affected by the average of each clip. At the same time, simply pooling the video in the time dimension may cause redundant or even invalid information in the video. In addition, for the second method, each action needs to be annotated in advance to divide the dataset into sub-stages. Changing a dataset requires re-annotation. The time for preliminary preparation is cumbersome and not universal. Although it can achieve good performance, it will cause a huge amount of computation. Summary of the Invention

[0005] The purpose of the present invention is to capture the coherence between actions to obtain the full dynamics of the complete action, which is conducive to evaluating the details of the action and increasing the reliability of the action quality assessment model. Since the features of each stage are independent of each other and there is a lack of connection between the fragments, the present invention proposes a fragment feature using an information interaction decoder to enhance the information between the fragment contexts within the clip. At the same time, the present invention proposes a lightweight adaptive weight allocation module that assigns different weights to each fragment. It is suitable for data sets with different actions, has excellent generalization ability, a small number of parameters and is easy to implement.

[0006] The present invention discloses an uncertainty action quality assessment method based on video interaction enhancement and proposes a new action quality assessment model: a self-attention adaptive weight allocation model. The overall framework is to extract feature sequences from video clips, explore the information interaction between video clips through clip features using an information interaction decoder and assign adaptive weights to the clips. Finally, an uncertainty-aware score distribution learning network is used to learn the predicted score probability, and MSE is used as the loss function. The loss function is only used in the training step to guide the model's predicted score to be close to the true score.

[0007] The specific implementation of the present invention includes the following steps:

[0008] Step (1). Obtain a single-person action quality assessment dataset, pre-process the dataset, and annotate the video;

[0009] Step (1.1). Obtain a single-person action quality assessment dataset, use Kinect to record the video and set the resolution to 720p and FPS to 25 frames;

[0010] Step (1.2): Preprocess the motion video and cut out redundant videos that are irrelevant to the final score, such as the preparatory phase before the start of the action and the waiting phase after the end of the action, retaining only the duration of each video from the start pose moment to the end pose moment;

[0011] Step (1.3). Filter valid videos by removing incomplete videos and annotating the scores of the dataset according to the actual evaluation scenario;

[0012] Step (2). The entire video sequence Divide into n segments, each video clip contains the same number of frames, representing continuous action snapshots, and extract features from different segments one by one to obtain clip-level features

[0013] Step (2.1). For a video with a total number of frames T, Y = {X1, X2, ...X T}, the entire video sequence Divided into n segments, each video clip contains the same number of frames, representing consecutive snapshots of the action;

[0014] Step (2.2). Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video clip to obtain clip-level features

[0015] Step (3). Use the information interaction decoder to enhance the information between the fragment contexts in the clip, and obtain the fragment information interaction enhanced features

[0016] Step (3.1). The information interaction decoder module is implemented through the self-attention mechanism. The query, key and value are all linear projections obtained from the N*1024 dimensional segment features, where N is the number of segments, after linear transformation: in, Represents query, key and value respectively;

[0017] Step (3.2). Then measure the relevance between video clips by the dot product similarity between corresponding query key pairs: in is the scaling factor, which makes the operation faster;

[0018] Step (3.3). Finally, the features of fragment information interaction enhancement It can be expressed as softmax(A n )·V n ;

[0019] Step (4). Each segment goes through an adaptive weight assignment module, which assigns different weights W to each segment. n , and the corresponding features Multiply to get the final eigenvector

[0020] Step (4.1). Pass all clip-level features through the combination of fully connected layers: connection layer 1-ReLU layer-fully connected layer 2-Softmax; among them, FC1 compresses features; ReLU activation function increases the nonlinearity of features; FC2 compresses the number of features to 1; Softmax function generates the weight W of the features n ;

[0021] Step (4.2). Adaptive weight W n Clip-level features enhanced by interaction with segment information After multiplication, add them according to the time dimension to get the final feature The formula is:

[0022] Step (5). The weighted features are learned through uncertainty perception score distribution to predict the final score probability. The score with the highest probability is the final quality assessment score.

[0023] Step (5.1). First, the characteristics After passing through the probability encoder Q, a random score variable s is obtained. This variable represents the score probability and obeys the Gaussian distribution: The parameters and The mean and variance of the features obtained after being encoded by the neural network can quantify the action score and its uncertainty. Based on this, they are sampled to output the predicted score;

[0024] Step (5.2). To ensure that encoder training is feasible and the score distribution sampling process is differentiable, we apply the reparameterization trick to the samples in the distribution instead of sampling directly from the score distribution. We choose the standard normal distribution (location = 0, scale = 1) as the auxiliary variable ε. Finally, according to the output of the independent sampling random variable ε, the mean parameter and variance parameters Calculating prediction scores

[0025] Step (5.3). For each training, the loss function is as follows: in s n and F v Represent the predicted score, the true score, and the video features, respectively. This loss function calculates the difference between the predicted score and the true score of each iteration of the neural network, thereby guiding the next step of training to make the predicted score close to the true score.

[0026] Compared with previous motion quality assessment methods, the present invention has the following benefits: 1. The segment features of the present invention use an information interaction decoder to enrich the information between video context segments, solving the problem of independence between segments, thereby capturing and enriching the relationship between different segments. 2. The present invention introduces a lightweight adaptive weight allocation module to assign adaptive weights to segments with a smaller number of parameters, quantifying the impact of segments on the final score, and better understanding the motion performance in the video. 3. The present invention uses an uncertainty-aware score distribution learning network to predict the distribution of scores rather than directly predicting scores, thereby improving the accuracy of motion quality score prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of the present invention.

[0028] Figure 2 This is an architecture diagram of the uncertain action quality assessment method based on video interaction enhancement of the present invention.

[0029] Figure 3 Structural diagram of the information interaction decoding process of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] This paper proposes an uncertainty action quality assessment method based on video interaction enhancement and proposes a new action quality assessment model: self-attention adaptive weight allocation model, such as Figure 1 The overall framework consists of a feature extraction module, an information interaction decoder, an adaptive weight assignment module, and an uncertainty score regression module. After extracting feature sequences from video clips, the information interaction decoder uses the segment features to explore the information interactions between video clips and assign adaptive weights to the segments. Finally, the network is trained using uncertainty-aware score distributions to learn the probability of predicted scores. The mean square error (MSE) is used as the loss function. This loss function is only used during training to guide the model's predicted scores closer to the true scores.

[0032] Next, the implementation steps will be described in detail with reference to the accompanying drawings.

[0033] Step (1). Obtain a single-person action quality assessment dataset, pre-process the dataset, and annotate the video;

[0034] Step (1.1). Obtain a single-person action quality assessment dataset, use Kinect to record the video and set the resolution to 720p and FPS to 25 frames;

[0035] Step (1.2): Preprocess the motion video and cut out redundant videos that are irrelevant to the final score, such as the preparatory phase before the start of the action and the waiting phase after the end of the action, retaining only the duration of each video from the start pose moment to the end pose moment;

[0036] Step (1.3). Filter valid videos by removing incomplete videos and annotating the scores of the dataset according to the actual evaluation scenario;

[0037] Step (2). The entire video sequence Divide into n segments, each video clip contains the same number of frames, representing continuous action snapshots, and extract features from different segments one by one to obtain clip-level features

[0038] Step (2.1). For a video with a total number of frames T, Y = {X1, X2, ...X T}, the entire video sequence Divided into n segments, each video clip contains the same number of frames, representing consecutive snapshots of the action;

[0039] Step (2.2). Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video clip to obtain clip-level features

[0040] Step (3). Use the information interaction decoder to enhance the information between the fragment contexts in the clip, and obtain the fragment information interaction enhanced features

[0041] Step (3.1). The information interaction decoder module is implemented through the self-attention mechanism. The query, key and value are all linear projections obtained from the N*1024 dimensional segment features, where N is the number of segments, after linear transformation: in, Represents query, key and value respectively;

[0042] Step (3.2). Then measure the relevance between video clips by the dot product similarity between corresponding query key pairs: in is the scaling factor, which makes the operation faster;

[0043] Step (3.3). Finally, the features of fragment information interaction enhancement It can be expressed as softmax(A n )·V n ;

[0044] Step (4). Each segment goes through an adaptive weight assignment module, which assigns different weights W to each segment. n , and the corresponding features Multiply to get the final eigenvector

[0045] Step (4.1). Pass all clip-level features through the combination of fully connected layers: fully connected layer 1-ReLU layer-fully connected layer 2-Softmax; among them, FC1 compresses features; ReLU activation function increases the nonlinearity of features; FC2 compresses the number of features to 1; Softmax function generates the weight W of the features n ;

[0046] Step (4.2). Adaptive weight W nClip-level features enhanced by interaction with segment information After multiplication, add them according to the time dimension to get the final feature The formula is:

[0047] Step (5). The weighted features are learned through uncertainty perception score distribution to predict the final score probability. The score with the highest probability is the final quality assessment score.

[0048] Step (5.1). First, the characteristics After passing through the probability encoder Q, a random score variable s is obtained. This variable represents the score probability and obeys the Gaussian distribution: The parameters and They are the mean and variance obtained after the features are encoded by the neural network, which can quantify the action score and its uncertainty; on this basis, they are sampled to output the predicted score;

[0049] Step (5.2). To ensure that encoder training is feasible and the score distribution sampling process is differentiable, we apply the reparameterization trick to the samples in the distribution instead of sampling directly from the score distribution. We choose the standard normal distribution (location = 0, scale = 1) as the auxiliary variable ε. Finally, according to the output of the independent sampling random variable ε, the mean parameter and variance parameters Calculating prediction scores

[0050] Step (5.3). For each training, the loss function is as follows: in s n and F v Represent the predicted score, the true score, and the video features, respectively. This loss function calculates the difference between the predicted score and the true score of each iteration of the neural network, thereby guiding the next step of training to make the predicted score close to the true score.

[0051] Preferably, the feature extraction module Video Swin Transformer (VST) described in step (2.2) uses weights pre-trained on the kinetics600 dataset.

[0052] Preferably, the number of heads of the information interaction decoder in step (3) is 8.

[0053] Preferably, the number of combined layers of the fully connected layers described in step (4.1) is 3.

[0054] The uncertainty action quality assessment method based on video interaction enhancement described in the above steps (1)-(5) is applied to two specific examples below, and its technical effect is further demonstrated through embodiments.

[0055] Example:

[0056] The steps of this embodiment are the same as those of the specific implementation method, so they will not be repeated here. The following is a demonstration of some of the experimental implementation processes and implementation results:

[0057] Data Sources: This example uses two datasets: Rhythmic Gymnastic and Fis-V. The Rhythmic Gymnastic dataset contains four rhythmic gymnastics movements: hoop, ball, club, and ribbon, with 250 videos each, for a total of 1,000 videos. Each video is approximately 1 minute and 35 seconds long and runs at 25 frames per second. Each video is annotated with three scores: difficulty, execution score, and total score, assigned by the judges on-site. 200 videos of each gymnastics type are used as the training set, and 50 videos are used as the test set. The Fis-V dataset is derived from women's single short program competition videos from high-quality international figure skating competitions from the past decade. Each video is approximately 2 minutes and 50 seconds long, with 4,300 frames and a frame rate of approximately 25 frames per second. Each video is annotated with two labels: the total element score (TES) and the total program composition score (PCS), provided by the judges on-site. The Fis-V dataset contains 500 videos, 400 of which are used for training and the remaining 100 for testing.

[0058] In order to verify the technical effect of the technical solution of the present invention, an indicator commonly used to test the effect of action quality evaluation is selected: Spearman rank correlation coefficient ρ to measure the performance of the detection. The specific definition is as follows:

[0059]

[0060] Where x and y represent the ranking of the two movements, respectively. The Spearman rank correlation coefficient has a score between -1 and 1, and the closer the absolute value is to 1, the higher the correlation. In addition, the cross-category average correlation (cross-category refers to movement type in Rhythmic Gymnastic and score type in Fis-V) is calculated from the Spearman rank correlation coefficient for each category using Fisher's z-score:

[0061]

[0062] Average correlation across classes

[0063] where ρ i is the rank correlation coefficient of the category, n is the category. The average correlation across categories ρavg The closer it is to 1, the better the effect.

[0064]

[0065] Table 1 Comparison of prediction results

[0066] SRC is the Spearman rank correlation coefficient, feature extraction is the feature extraction method of the dataset, ball, stick, circle, and ribbon are the action categories of Rhythm Gymnastic, TES and PCS are the score categories of the Fis-v dataset, and average 1 and average 2 are the average values ​​of the two datasets in different action categories or score categories. SVR, MS-LSTM, ACTION-NET, and GDLT are all compared methods in this paper.

[0067] The test results are shown in Table 1. The results show that the prediction effect of the method proposed in the present invention is: the Spearman rank correlation coefficient is significantly better than other methods, verifying the effectiveness of the method of the present invention.

[0068] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. Uncertain action quality assessment method based on video interaction enhancement, characterized in that: The following steps are involved: Step 1. Obtain a single-person action quality assessment dataset, pre-process the dataset, and annotate the video; Step 2. Convert the entire video sequence Divide into n segments, each video clip contains the same number of frames, representing continuous action snapshots, and extract features from different segments one by one to obtain clip-level features Step 3. Use the information interaction decoder to enhance the information between the fragment contexts in the clip and obtain the features enhanced by the fragment information interaction. Step 4. Each segment passes through an adaptive weight assignment module, which assigns different weights W to each segment. n , and the corresponding features Multiply them together to get the final eigenvector Step 5. Finally, the weighted features are subjected to uncertainty-aware score distribution learning to predict the final score probability. The score with the highest probability is the final quality assessment score.

2. The uncertain action quality assessment method based on video interaction enhancement according to claim 1 is characterized in that: Step 1 The specific process is as follows: 1.1 Obtain a single-person action quality assessment dataset, use Kinect to record the video and set the resolution to 720p and the FPS to 25 frames; 1.2 Preprocess the motion video and cut out the redundant videos that are irrelevant to the final score, such as the preparatory stage before the action and the waiting stage after the action. Only the duration of each video from the start pose to the end pose is retained; 1.3 Filter valid videos by removing some incomplete videos and annotating the scores of the dataset according to the actual evaluation scenario.

3. The uncertain action quality assessment method based on video interaction enhancement according to claim 2 is characterized in that: Step 2: 2.1 For a video with a total number of frames T, Y = {X1, X2, ...X T }, the entire video sequence Divided into n segments, each video clip contains the same number of frames, representing consecutive snapshots of the action; 2.2 Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video clip to obtain clip-level features 4. The uncertain action quality assessment method based on video interaction enhancement according to claim 3 is characterized in that: The specific process of step 3 is as follows: 3.1 Information Interaction The decoder module is implemented through the self-attention mechanism; the query, key and value are all linear projections obtained from the N*1024-dimensional fragment features, where N is the number of fragments, after linear transformation: in, Represents query, key and value respectively; 3.2 Then measure the correlation between video clips by the dot product similarity between corresponding query key pairs: in is the scaling factor, which makes the operation faster; 3.3 Finally, features enhanced by fragment information interaction It can be expressed as softmax(A n )·V n .

5. The uncertain action quality assessment method based on video interaction enhancement according to claim 4 is characterized in that: Step 4: The specific process is as follows: 4.1 All clip-level features are passed through the combination of fully connected layers: fully connected layer 1-ReLU layer-fully connected layer 2-Softmax; among them, FC1 compresses features; ReLU activation function increases the nonlinearity of features; FC2 compresses the number of features to 1; Softmax function generates the weight W of the features n ; 4.2 Adaptive Weight W n Clip-level features enhanced interactively with segment information After multiplication, add them together according to the time dimension to get the final feature The formula is:

6. The uncertain action quality assessment method based on video interaction enhancement according to claim 5 is characterized in that: Step 5: The specific process is as follows: 5.1 First Features After passing through the probability encoder Q, a random score variable s is obtained, which represents the score probability and obeys the Gaussian distribution: The parameters and They are the mean and variance obtained after the features are encoded by the neural network, which can quantify the action score and its uncertainty; on this basis, they are sampled to output the predicted score; 5.2 To ensure that encoder training is feasible and the score distribution sampling process is differentiable, we apply the reparameterization technique to samples in the distribution instead of sampling directly from the score distribution. We choose the standard normal distribution (location = 0, scale = 1) as the auxiliary variable ε. Finally, according to the output of the independent sampling random variable ε, the mean parameter and variance parameter Calculating prediction scores 5.3 For each training, the loss function is as follows: in s n and F v Represent the predicted score, the true score and the video feature respectively. This loss function calculates the difference between the predicted score and the true score of each iteration of the neural network, thereby guiding the next step of training to make the predicted score close to the true score.

Images