Semi-supervised action quality evaluation method and system based on vector quantization and decoupling representation

Abstract

The invention discloses a semi-supervised motion quality assessment method and system based on vector quantization and decoupling characterization, and relates to the field of motion quality assessment, and the method comprises the steps: obtaining an original feature sequence corresponding to input video data; performing nonlinear transformation on the original feature sequence to respectively obtain a feature sequence corresponding to the sub-action category and a feature sequence corresponding to the sub-action execution condition; performing discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain a first vector quantization result; performing continuous vector quantization on the feature sequence corresponding to the sub-action execution condition to obtain a second vector quantization result; and splicing the first vector quantization result and the second vector quantization result along the feature dimension and taking the spliced result as input of a pre-trained action quality prediction module to obtain an action quality evaluation result. According to the method, by decoupling the category features and the execution condition features of the sub-actions, the interpretability and robustness of the system are effectively improved, and higher prediction accuracy and expansibility are provided.

Description

Technical Field

[0001] The present invention relates to the field of action quality assessment, and particularly to a semi-supervised action quality assessment method and system based on vector quantization and decoupled representation. Background Art

[0002] Action quality assessment is used to evaluate the completion quality of one or a series of actions. The target objects are one or a group of people performing actions, including but not limited to: athletes participating in sports, medical staff participating in training or practical skills examinations, operators in a factory production environment, etc. Action quality assessment aims to establish an automated qualitative or quantitative assessment method to overcome two main problems existing in the traditional assessment process:

[0003] 1) The assessment process depends on the participation of experts with rich experience in a specific field, which requires high human and economic resources.

[0004] 2) There are differences in the understanding of rules and the actual evaluation process among different evaluation experts, resulting in difficulties in making parallel comparisons between evaluation results.

[0005] Existing action quality assessment methods are mainly constructed based on machine learning methods. By collecting a batch of original data on the execution of actions (typically video recordings of people performing actions), and the corresponding labels, that is, the on-site or later expert review results (presented in the form of scores within a specific range, or grades such as excellent, good, medium, and poor for execution), a model is designed and trained to fit the real results. Among them, the cost of collecting and uniformly annotating labels is significantly higher than that of collecting videos because a large number of professionals need to be involved in interpreting the rules and standardizing the annotation forms.

[0006] From the perspective of the utilization of training data, existing methods can be divided into two categories: fully supervised learning and semi-supervised learning. Supervised Learning requires all samples to have a "video - label" pair. Therefore, the main disadvantage of this type of method is that it is difficult to balance the contradiction between the cost of collecting labeled data and training a model with good performance and a large number of parameters, and it cannot effectively utilize the more easily collected video data; Semi-supervised Learning uses a small number of "video - label" pairs and the remaining large number of unlabeled videos as training samples, which can more effectively utilize the samples lacking labels, establish connections with the labeled samples to improve the generalization ability and data utilization efficiency of the model, and is convenient for improving the number of model parameters and performance by expanding the data scale.

[0007] Specifically, the existing action quality assessment methods based on unsupervised learning include the following categories:

[0008] (1) The first prior art uses teacher-student distillation. The input of the teacher network is labeled training samples. After using an additional object-detection model to locate moving people in the video and then extracting features, the input of the student network is unlabeled samples, which directly extract features from the untreated video, and narrow the distribution difference between the two branches of features to obtain a robust feature representation. Based on the assumption that "the sub-action sequences of the same type of movement (sub-actions are the basic sequence units or execution stages that make up the evaluated action. For example, in the process of a diving 'action', it can be subdivided into starting, vertical jump in place, mid-air rotation, entering the water, etc.'sub-actions') remain unchanged", group contrastive learning is used for sub-action localization and video segment cutting to further improve the interpretability and stability of the model. The disadvantages of this method are:

[0009] 1) The overall network structure is too complex and relies on an additional target network, which has no significant effect in aligning the features of the teacher-student networks, and will affect the overall performance of its method when the performance of the object-detection model fluctuates;

[0010] 2) The assumption about the action sequence is too strong and cannot be applied in the evaluation of some non-specific sequence actions (for example, in figure skating, the athlete can freely choose the order of performing two types of sub-actions,'spin' and 'jump'. In this scenario, the premise of the network design does not hold absolutely), which will lead to a decline in performance.

[0011] (2) The second prior art constructs a self-supervised learning objective by predicting masked video frames to improve the robustness of feature representation, and narrows the distribution distance between labeled samples and unlabeled samples through adversarial learning. The disadvantages of this method are:

[0012] 1) The training process is relatively complex. The objectives of self-supervised learning and adversarial learning are greatly affected by hyperparameters in the actual training process and are prone to the situation of model collapse;

[0013] 2) It fails to make full use of unlabeled data, only plays its role in stabilizing feature representation, and lacks the potential to further improve the model prediction performance by using the correlation between unlabeled samples and labeled samples.

[0014] (3) Prior Art III adds a branch to the traditional teacher-student network to form a "Teacher-Reference-Student" structure. The teacher branch and the student branch take labeled samples and unlabeled samples as inputs respectively, while the reference branch calculates the differences between "labeled-unlabeled" sample pairs through cross-attention to provide additional pseudo-labels to the student network as supervision information for updating network parameters. The disadvantages of this method are: it does not fully utilize the characteristic of consistent feature distribution to optimize the feature expression process, and similar to Prior Art II, it lacks the potential to improve the model prediction performance by leveraging its correlation with labeled samples.

[0015] In summary, the existing action quality assessment methods have the following disadvantages:

[0016] 1. Existing methods regard action quality assessment as overall modeling, lacking the distinction of subtask details (such as sub-action categories and execution situations), resulting in the model being difficult to expand and lacking interpretability.

[0017] 2. Existing methods are easily affected by hyperparameter changes during training, leading to a decline or collapse in model performance. Especially in multi-stage training, the model structure is complex and difficult to adjust.

[0018] 3. Existing methods usually use unlabeled data for feature alignment or pseudo-label generation, lacking effective utilization of unlabeled data and easily introducing noisy data to affect the training results. Summary of the Invention

[0019] In view of the above deficiencies in the prior art, a semi-supervised action quality assessment method and system based on vector quantization and decoupled representation provided by the present invention solve the problem that existing methods regard action quality assessment as overall modeling, lacking the distinction of subtask details, resulting in the model being difficult to expand and lacking interpretability.

[0020] To achieve the above invention purpose, the technical solution adopted by the present invention is:

[0021] Provide a semi-supervised action quality assessment method based on vector quantization and decoupled representation, which includes:

[0022] Extract features from the input video data through a pre-trained primary video feature extraction module to obtain an original feature sequence;

[0023] Perform a non-linear transformation on the original feature sequence through a pre-trained first multi-layer perceptron to obtain a feature sequence corresponding to sub-action categories;

[0024] The original feature sequence is non-linearly transformed by a pre-trained second multi-layer perceptron to obtain a feature sequence corresponding to the sub-action execution situation;

[0025] The feature sequence corresponding to the sub-action category is discretely vector quantized by a pre-trained discrete vector quantization module to obtain a first vector quantization result;

[0026] The feature sequence corresponding to the sub-action execution situation is continuously vector quantized by a pre-trained continuous vector quantization module to obtain a second vector quantization result;

[0027] The first vector quantization result and the second vector quantization result are concatenated along the feature dimension and used as the input of a pre-trained action quality prediction module to obtain an action quality evaluation result.

[0028] Further, the specific method for extracting features from the input video data by a pre-trained primary video feature extraction module includes:

[0029] The input video data is segmented into slices with 16 frames each, and each slice is feature-extracted by a pre-trained primary video feature extraction module, thereby obtaining an original feature sequence formed by the features corresponding to all slices; wherein, when the last slice has less than 16 frames, it is padded with a zero tensor.

[0030] Further, the dimensions of the feature sequence corresponding to the sub-action category, the dimensions of the feature sequence corresponding to the sub-action execution situation, and the dimensions of the original feature sequence are all the same.

[0031] Further, the specific method for discretely vector quantizing the feature sequence corresponding to the sub-action category to obtain a first vector quantization result includes:

[0032] Randomly initialize a set containing N E embeddings Calculate respectively the distance between each embedding in and the feature sequence corresponding to the sub-action category, and select the embedding with the smallest distance as the first vector quantization result.

[0033] Further, the specific method for continuously vector quantizing the feature sequence corresponding to the sub-action execution situation to obtain a second vector quantization result is:

[0034] Randomly initialize a set containing N E embeddings

[0035] Obtain the weights of each embedding in by calculating the cosine similarity and normalize them, and its expression is:

[0036] ​

[0037] where w j is the j-th embedding e j in the normalized weight; is the feature sequence corresponding to the sub-action execution; e k represents the k-th embedding; <.> represents calculating the cosine similarity;

[0038] Multiply each embedding in by its corresponding normalized weight and sum them to obtain the second vector quantization result, and its expression is:

[0039]

[0040] where is the second vector quantization result.

[0041] Furthermore, the pre-trained action quality prediction module includes a network structure with two layers of Transformer-Encoder and a third multi-layer perceptron;

[0042] The network structure with two layers of Transformer-Encoder is used to globally aggregate the feature data input to the pre-trained action quality prediction module, and take the first element of the output sequence after global aggregation as the input of the multi-layer perceptron;

[0043] The third multi-layer perceptron is used to evaluate the action quality according to the input data and obtain the action quality evaluation result.

[0044] Furthermore, it also includes:

[0045] Calculate the confidence based on the feature sequence corresponding to the sub-action category and the first vector quantization result to obtain the first confidence, and its expression is:

[0046]

[0047] where ξ D is the first confidence; is the first vector quantization result; is the feature sequence corresponding to the sub-action category; 〈.〉 represents calculating the cosine similarity; max(.) represents the maximum value function;

[0048] Randomly initialize a set E containing N Based on the feature sequence corresponding to the sub-action execution and the set calculate the confidence to obtain the second confidence, and its expression is:

[0049]

[0050] where w j is the normalized weight corresponding to the j-th embedding e j in the is the feature sequence corresponding to the sub-action execution; e k represents the k-th embedding; <.> represents calculating the cosine similarity; ξ C is the second confidence; N E is the total number of randomly initialized embeddings; log(.) represents the logarithm to the base 2;

[0051] Multiply the first confidence and the second confidence element by element and take the average to obtain the confidence of the action quality evaluation result of the current input video data, and its expression is:

[0052]

[0053] where is the confidence of the action quality evaluation result of the current input video data; M is the sequence length of the first confidence and the second confidence; is the i-th element in the first confidence; is the i-th element in the second confidence.

[0054] Provide a system for a semi-supervised action quality evaluation method based on vector quantization and decoupled representation, which includes a pre-trained primary video feature extraction module, a pre-trained first multi-layer perceptron, a pre-trained second multi-layer perceptron, a pre-trained discrete vector quantization module, a pre-trained continuous vector quantization module, and a pre-trained action quality prediction module; where:

[0055] The pre-trained primary video feature extraction module is used to extract features from the input video data to obtain an original feature sequence;

[0056] The pre-trained first multi-layer perceptron is used to perform a non-linear transformation on the original feature sequence to obtain a feature sequence corresponding to the sub-action category;

[0057] The pre-trained second multi-layer perceptron is used to perform a non-linear transformation on the original feature sequence to obtain a feature sequence corresponding to the sub-action execution;

[0058] The pre-trained discrete vector quantization module is used to perform discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain a first vector quantization result;

[0059] The pre-trained continuous vector quantization module is used to perform continuous vector quantization on the feature sequence corresponding to the sub-action execution to obtain a second vector quantization result;

[0060] A pre-trained action quality prediction module, which is used to splice the first vector quantization result and the second vector quantization result along the feature dimension, and perform action quality evaluation based on the splicing result to obtain an action quality evaluation result.

[0061] Furthermore, the pre-training method of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module and the action quality prediction module includes two stages. Among them, the first stage includes the following operations:

[0062] Prepare for the first stage of training, including:

[0063] Divide the labeled training set, unlabeled training set, and validation set; initialize the system hyperparameters, number of training epochs, learning rate, optimizer, and early stopping conditions;

[0064] Perform the first stage of iterative training, including:

[0065] Take samples from the labeled training set in batches and input them into the system, obtain the total system loss value and perform gradient backpropagation, and update the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module and the action quality prediction module through the optimizer;

[0066] Take samples from the unlabeled training set in batches and input them into the system, obtain the total system loss value and perform backpropagation, and update the parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron and the discrete vector quantization module through the optimizer;

[0067] Take samples from the validation set in batches and input them into the system, calculate the performance index of the system after obtaining the action quality prediction results of all validation samples and the true action quality;

[0068] When the first stage reaches the maximum number of training epochs, or the performance index of the system obtained from the validation set meets the early stopping conditions, stop the first stage of iterative training and enter the second stage; the second stage includes the following operations:

[0069] Prepare for the second stage of training, including:

[0070] Set the number of training epochs, learning rate, optimizer, pseudo-label screening confidence threshold, and early stopping conditions;

[0071] Perform the second stage of iterative training, including:

[0072] Samples are taken from the labeled training set in batches and input into the system. The total loss value of the system is obtained and gradient backpropagation is performed. The parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module, and the action quality prediction module are updated through the optimizer;

[0073] Samples are taken from the unlabeled training set in batches and input into the system to obtain the corresponding action quality prediction results and confidence levels; the unlabeled training samples corresponding to the confidence levels higher than the pseudo-label screening confidence threshold are added to the labeled training set;

[0074] Samples are taken from the validation set in batches and input into the system. After obtaining the action quality prediction results of all validation samples, the performance indicators of the system are calculated with the true action quality;

[0075] When the maximum number of training epochs in the second stage is reached, or the performance indicators of the system obtained from the validation set meet the early stopping conditions, the iterative training in the second stage is stopped, and the pre-training of the system is completed.

[0076] Furthermore, the calculation method of the total loss value of the system includes:

[0077] The triplet loss is calculated based on the feature sequence corresponding to the sub-action category output by the first multi-layer perceptron and the feature sequence corresponding to the sub-action execution situation output by the second multi-layer perceptron. Its expression is:

[0078]

[0079] where D + and D - are both intermediate parameters; is the j-th element in the feature sequence corresponding to the sub-action category; is the k-th element in the feature sequence corresponding to the sub-action category; dist(.) represents the dist function; max j≠k (.) represents taking the maximum value; and are the j-th element and the k-th element in the feature sequence corresponding to the sub-action execution situation respectively; min j,k (.) represents taking the minimum value; α tri is a hyperparameter used to control the loss calculation; is the triplet loss;

[0080] The discrete vector quantization loss is calculated. Its expression is:

[0081]

[0082] where is the discrete vector quantization loss; The feature sequence corresponding to the sub - action category; Is the first vector quantization result; sg(.) represents the gradient truncation operation; β is the proximity rate control hyperparameter;

[0083] Randomly initialize a set containing N E embeddings Calculate the continuous vector quantization loss, and its expression is:

[0084]

[0085] where represents taking out a single embedding one by one from the set based on the L2 distance from and taking the embedding corresponding to the minimum distance, that is The feature sequence corresponding to the sub - action execution situation; e j is the j - th embedding in; is the continuous vector quantization loss;

[0086] Calculate the prediction loss, and its expression is:

[0087]

[0088] where is the prediction loss; is the action quality evaluation result of the current input video data; y is the true action quality result of the current input video data;

[0089] Calculate the total loss:

[0090]

[0091] where is the above - mentioned pre - training; λ1, λ2, λ3 and λ4 are weight hyperparameters; when the current input video sample is an unlabeled sample, λ1 = 0.

[0092] The beneficial effects of the present invention are:

[0093] 1. By decoupling the category features and execution situation features of sub - actions, the present invention effectively improves the interpretability and robustness of the system, and provides higher prediction accuracy and scalability.

[0094] 2. By simplifying the model structure and optimization objective, the present invention reduces the dependence on other pre - trained network modules, and adopts a simplified two - stage training strategy. In the first stage, some network structures and training parameters are adjusted. After ensuring the system stability, it enters the second stage, thereby reducing the trial - and - error cost.

[0095] 3. The present invention directly utilizes the feature quantization representation of labeled data, reducing the instability in system training. In addition, the present invention generates confidence through a parameter-free method (based on similarity and information entropy) to screen pseudo-label samples, avoiding the introduction of noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 is a schematic flow chart of the method;

[0097] Figure 2 is a schematic structural diagram of the system;

[0098] Figure 3 is a schematic training flow chart of the system. DETAILED DESCRIPTION OF THE INVENTION

[0099] The following describes the specific embodiments of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions made using the concept of the present invention are within the scope of protection.

[0100] As Figure 1 shown, the semi-supervised action quality assessment method based on vector quantization and decoupled representation includes:

[0101] Step A: Extract features from the input video data through a pre-trained primary video feature extraction module to obtain an original feature sequence;

[0102] Step B: Perform a non-linear transformation on the original feature sequence through a pre-trained first multi-layer perceptron to obtain a feature sequence corresponding to the sub-action category;

[0103] Step C: Perform a non-linear transformation on the original feature sequence through a pre-trained second multi-layer perceptron to obtain a feature sequence corresponding to the sub-action execution situation;

[0104] Step D: Perform discrete vector quantization on the feature sequence corresponding to the sub-action category through a pre-trained discrete vector quantization module to obtain a first vector quantization result;

[0105] Step E: Perform continuous vector quantization on the feature sequence corresponding to the sub-action execution situation through a pre-trained continuous vector quantization module to obtain a second vector quantization result;

[0106] Step F: Concatenate the first vector quantization result and the second vector quantization result along the feature dimension and use them as the input of a pre-trained action quality prediction module to obtain an action quality assessment result.

[0107] In this embodiment, the first multi-layer perceptron and the second multi-layer perceptron constitute a decoupled feature module.

[0108] In this embodiment, each frame of the video data to be evaluated is scaled to a unified resolution of 1920×1080 and represented in the RGB channel format. Then, central cropping is performed to change the video frame resolution to 1080×1080. Finally, scaling is performed to change the video resolution to 224×224, obtaining the video data to be input into the primary video feature extraction module, so as to complete the resolution preprocessing of the video data to be evaluated.

[0109] In this embodiment, the specific method for extracting features from the input video data by the pre-trained primary video feature extraction module includes:

[0110] For the video data input into the primary video feature extraction module, it is segmented into slices with every 16 frames as a slice. The pre-trained primary video feature extraction module is used to extract features from each slice, thereby obtaining an original feature sequence formed by the features corresponding to all slices. Among them, when the last slice has less than 16 frames, it is padded with a tensor of all zeros.

[0111] In the specific implementation process, the primary video feature extraction module is implemented based on an action recognition network. When the action recognition network is used in the present invention, its last classification linear layer is replaced by a multi-layer perceptron, and the weights of the action recognition network part during the training process are frozen, and only the parameters of the newly added multi-layer perceptron are updated. The unchanged action recognition network can be an Inflated 3D Convolutional Network (I3D), a Pseudo 3D Convolutional Network (P3D), a Video Swin Transformer (VST), etc.

[0112] In this embodiment, the dimensions of the feature sequence corresponding to the sub-action category, the dimensions of the feature sequence corresponding to the sub-action execution situation, and the dimensions of the original feature sequence are all M.

[0113] In this embodiment, the specific method for performing discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain the first vector quantization result includes:

[0114] Randomly initialize a set containing N E embeddings Calculate respectively the distances between each embedding in and the feature sequence corresponding to the sub-action category, and select the embedding with the smallest distance as the first vector quantization result. The distance between each embedding and the feature sequence corresponding to the sub - action category can adopt the L2 distance, and the corresponding expression is:

[0115]

[0116] where is the first vector quantization result; is the feature sequence corresponding to the sub - action category; e i is the set the i - th embedding in it.

[0117] In this embodiment, the operation of the continuous vector quantization module is similar to that of the discrete vector quantization module, including N E randomly initialized embeddings whose set is denoted as The difference is that the execution situation of the sub - action cannot directly adopt fixed vector substitution to reflect its basic property of continuity (for example, evaluating whether a sub - action is executed in place may involve a continuous change process from "very poor completion quality" to "fair completion quality" and then to "very good completion situation", and it is not appropriate to use fixed discrete points to represent its feature vectors). Therefore, there are the following differences in the specific calculation method: the output is the result linearly represented by all the embeddings in, and the weight calculation is based on the result of calculating the cosine similarity between the input and each (and perform weight normalization). The more similar the embeddings are, the higher the weighted weight they contribute.

[0118] Based on this, the specific method for continuous vector quantization of the feature sequence corresponding to the execution situation of the sub - action to obtain the second vector quantization result includes:

[0119] Step E1: Randomly initialize a set E containing N

[0120] Step E2: Obtain the weight of each embedding in by calculating the cosine similarity and perform normalization. The expression is:

[0121]

[0122] where w j is the j - th embedding e j in after normalization; is the feature sequence corresponding to the execution situation of the sub - action; e k represents the k - th embedding; <.> represents calculating the cosine similarity;

[0123] Step E3: Multiply each embedding in by its corresponding normalized weight and sum them up to obtain the second vector quantization result, and its expression is:

[0124]

[0125] where is the second vector quantization result.

[0126] In this embodiment, the pre-trained action quality prediction module includes a network structure with two layers of Transformer-Encoder and a third multi-layer perceptron;

[0127] The network structure with two layers of Transformer-Encoder is used to globally aggregate the feature data input to the pre-trained action quality prediction module, and take the first element of the output sequence after global aggregation as the input of the multi-layer perceptron;

[0128] The third multi-layer perceptron is used to evaluate the action quality according to the input data and obtain the action quality evaluation result.

[0129] In the specific implementation process, in order to show the reliability of the evaluation results of the present method / system to the user, a confidence level is generated for the corresponding evaluation results for reference (for example, if the confidence level is seen to be too low, the user can choose not to accept this result or request manual intervention), and the method for generating the confidence level includes:

[0130] Step G1: Calculate the confidence level based on the feature sequence corresponding to the sub-action category and the first vector quantization result through cosine similarity (taking the non-negative result) to obtain the first confidence level, and its expression is:

[0131]

[0132] where ξ D is the first confidence level; is the first vector quantization result; is the feature sequence corresponding to the sub-action category; <.> represents calculating cosine similarity; max(.) represents the maximum value function;

[0133] Step G2: Randomly initialize a set containing N E embeddings Calculate the confidence level based on the feature sequence corresponding to the sub-action execution situation and the set to obtain the second confidence level, and its expression is:

[0134]

[0135] where w j is the j-th embedding e in j the corresponding normalized weight; is the feature sequence corresponding to the sub-action execution; e k represents the k-th embedding; <.> represents calculating the cosine similarity; ξ C is the second confidence; N E is the total number of randomly initialized embeddings; log(.) represents the logarithm with base 2;

[0136] In this step, the confidence is calculated based on the (normalized) negative information entropy. That is, when the input feature is similar to multiple embeddings, the information entropy increases accordingly, and the confidence decreases;

[0137] Step G3: Multiply the first confidence and the second confidence element by element and take the average to obtain the confidence of the action quality evaluation result of the current input video data. Its expression is:

[0138]

[0139] where is the confidence of the action quality evaluation result of the current input video data; M is the sequence length of the first confidence and the second confidence; is the i-th element in the first confidence; is the i-th element in the second confidence.

[0140] In this embodiment, as Figure 2 shown, a system based on a semi-supervised action quality evaluation method based on vector quantization and decoupled representation is also provided. The system includes a pre-trained primary video feature extraction module, a pre-trained first multi-layer perceptron, a pre-trained second multi-layer perceptron, a pre-trained discrete vector quantization module, a pre-trained continuous vector quantization module, and a pre-trained action quality prediction module; where:

[0141] The pre-trained primary video feature extraction module is used to extract features from the input video data to obtain the original feature sequence;

[0142] The pre-trained first multi-layer perceptron is used to perform a non-linear transformation on the original feature sequence to obtain the feature sequence corresponding to the sub-action category;

[0143] The pre-trained second multi-layer perceptron is used to perform a non-linear transformation on the original feature sequence to obtain the feature sequence corresponding to the sub-action execution;

[0144] The pre-trained discrete vector quantization module is used to perform discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain the first vector quantization result;

[0145] A pre-trained continuous vector quantization module for performing continuous vector quantization on the feature sequence corresponding to the sub-action execution situation to obtain a second vector quantization result;

[0146] A pre-trained action quality prediction module for concatenating the first vector quantization result and the second vector quantization result along the feature dimension, and performing action quality evaluation based on the concatenated result to obtain an action quality evaluation result.

[0147] In this embodiment, as Figure 3 shown, the pre-training methods of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module, and the action quality prediction module (i.e., the pre-training method of the system) include two stages. Among them, the first stage includes the following operations:

[0148] Prepare for the first-stage training, including the steps of:

[0149] S201. Set the global random seed, divide the labeled training set, unlabeled training set, and validation set; set the hyperparameters of the system network structure, such as the number of multi-layer perceptron layers, the hidden layer dimension, the decoupled feature representation dimension, the activation function, the number of embeddings for vector quantization, etc.;

[0150] S202. Set the main hyperparameters of the first stage, including the number of training epochs, the learning rate, the optimizer, the early stopping condition, etc. The main purpose of the first stage is to obtain a stable feature representation. The labeled samples are used to directly optimize the network weights by reducing the gap between the predicted value and the true value in an end-to-end manner, and jointly optimize the decoupled representation and vector quantization parts with the unlabeled data, playing a role in aligning the feature expressions of the unlabeled samples with those of the labeled samples, improving the utilization efficiency of the unlabeled data, and enhancing the generalization ability of the model.

[0151] Perform the first-stage iterative training, including the steps of:

[0152] S203. Batch-sample from the labeled training set and input them into the system to obtain the action quality prediction result of the system. Obtain the total loss value of the system according to the label and perform gradient backpropagation, and update the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module, and the action quality prediction module through the optimizer;

[0153] S204. Batch-sample from the unlabeled training set and input them into the system to obtain the total loss value of the system and perform backpropagation, and update the parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, and the discrete vector quantization module through the optimizer;

[0154] S205. Take samples from the validation set in batches and input them into the system. After obtaining the action quality prediction results of all validation samples, calculate the performance metrics of the system (such as Spearman Correlation, Relative L2 Distance, etc.) with the true action quality;

[0155] S206. When the maximum number of training epochs in the first stage is reached, or the performance metrics of the system obtained from the validation set meet the early stopping condition, stop the iterative training in the first stage and enter the second stage; otherwise, return to step S203 to continue the iterative training in the first stage;

[0156] The second stage includes the following operations:

[0157] Prepare for the second stage of training, including the steps:

[0158] S207. Set the main hyperparameters for the second stage, including the number of training epochs, learning rate, optimizer, confidence threshold for pseudo-label screening (a fixed value or changing with the training epochs, for example, linearly decreasing from 1 to 0.5 as the training batches increase), early stopping condition (for example, the metrics do not exceed the historical best in several training loops, or the parameter gradient is less than the threshold), etc. The main purpose of this stage is to further optimize the system by introducing more reliable estimated results of unlabeled samples, but it is necessary to avoid interference with the feature expression part. Therefore, the learning rates of the networks of the primary video feature extraction module, decoupled feature module, discrete vector quantization module, and action quality prediction module need to be reduced.

[0159] Perform iterative training in the second stage, including the steps:

[0160] S208. Take samples from the labeled training set in batches and input them into the system. Obtain the total loss value of the system and perform gradient backpropagation. Update the parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module, and the action quality prediction module through the optimizer;

[0161] S209. Take samples from the unlabeled training set in batches and input them into the system. Obtain the corresponding action quality prediction results and confidence levels; add the unlabeled training samples with confidence levels higher than the pseudo-label screening confidence threshold to the labeled training set;

[0162] S210. Take samples from the validation set in batches and input them into the system. After obtaining the action quality prediction results of all validation samples, calculate the performance metrics of the system (such as Spearman Correlation, Relative L2 Distance, etc.) with the true action quality;

[0163] S211. When the second stage reaches the maximum number of training epochs, or the performance metrics of the system obtained from the validation set meet the early stopping conditions (for example, the metrics do not exceed the historical optimum in several training cycles, or the parameter gradient is less than the threshold), stop the iterative training in the second stage and complete the pre-training of the system. Otherwise, add all the samples in the labeled training set to the training set in the second stage, and return to step S208 to continue the iterative training in the second stage.

[0164] In this embodiment, the calculation method of the total system loss value includes:

[0165] Calculate the triplet loss based on the feature sequence corresponding to the sub-action category output by the first multi-layer perceptron and the feature sequence corresponding to the sub-action execution situation output by the second multi-layer perceptron. Its expression is:

[0166]

[0167] where D + and D - are both intermediate parameters; is the j-th element in the feature sequence corresponding to the sub-action category; is the k-th element in the feature sequence corresponding to the sub-action category; dist(.) represents the dist function; max j≠k (.) represents taking the maximum value; and are the j-th element and the k-th element in the feature sequence corresponding to the sub-action execution situation respectively; min j,k (.) represents taking the minimum value; α tri is a hyperparameter used to control the loss calculation; is the triplet loss;

[0168] Calculate the discrete vector quantization loss. Its expression is:

[0169]

[0170] where is the discrete vector quantization loss; is the feature sequence corresponding to the sub-action category; is the first vector quantization result; sg(.) represents the gradient clipping operation; β is the proximity rate control hyperparameter;

[0171] Randomly initialize a set containing N E embeddings Calculate the continuous vector quantization loss. Its expression is:

[0172]

[0173] where Denote taking out a single embedding from the set one by one, based on the L2 distance with and taking the embedding corresponding to the smallest distance, i.e., is the feature sequence corresponding to the execution situation of the sub-action; e j is the j-th embedding in is the continuous vector quantization loss;

[0174] Calculate the prediction loss, and its expression is:

[0175]

[0176] where is the prediction loss; is the action quality evaluation result of the current input video data; y is the true action quality result of the current input video data;

[0177] Calculate the total loss:

[0178]

[0179] where is the above-mentioned pre-trained; λ1, λ2, λ3, and λ4 are weight hyperparameters; when the current input video sample is an unlabeled sample, λ1 = 0.

[0180] In the specific implementation process of the present invention, performance comparisons between the present invention and various advanced methods are given on the commonly used data sets in three fields. First, the data sets are introduced respectively:

[0181] ① MTL-AQA data set: A multi-task data set for describing diving, containing 1412 samples. In the experiment, the action score corresponding to each sample is used as the label for training and testing.

[0182] ② rhythmic gymnastics (RG) data set: An artistic gymnastics data set containing four sub-data sets of competition events (respectively Ball, Clubs, Hoop, Ribbon), and each sub-data set contains 250 samples. In the experiment, the action score corresponding to each sample is used as the label for training and testing.

[0183] ③Fine Figure Skating (FineFS) Dataset: A multi-task figure skating dataset containing two sub-datasets for competition events (Short Program and Free Skating), with 729 and 438 samples respectively. In the experiment, the action scores corresponding to each sample were used as labels for training and testing. Specifically, there are two types of score labels, namely Program Component Score (PCS) and Technical Element Score (TES).

[0184] The metric for performance testing is the Spearman Correlation, which is used to measure the ranking correlation between two sets of data (i.e., the predicted score sequence and the true score sequence on the entire test sample set), and to measure the consistency (linear correlation) between the ranking relationship of scores among the predicted samples and the actual score ranking relationship. The higher the result, the more accurate the prediction, and the maximum value is 1.00. The specific experimental settings and the results of performance comparison are as follows:

[0185] (1) Performance comparison on the MTL-AQA dataset

[0186] For 1412 samples, 75% of the samples (1059) were randomly selected for model training, and the remaining 25% (353) were used to test the model performance. Compared with the previous scheme, training and testing were carried out under two supervised ratios, namely 40% (i.e., 424 samples in the training samples were used as labeled samples, and the remaining 635 were used as unlabeled samples) and 10% (i.e., 106 samples in the training samples were used as labeled samples, and the remaining 953 were used as unlabeled samples). The results of comparing this method with known advanced methods are shown in Table 1 below.

[0187] Table 1

[0188]

[0189]

[0190] (2) Performance comparison on the RG dataset

[0191] For the 4 sub-datasets, each containing 250 samples, the training and testing processes were independently carried out on each sub-dataset. The proportion of training samples was 75% (i.e., 188 samples), among which 40% (i.e., 75) of the samples were labeled, 60% (i.e., 113) of the samples were not labeled, and the remaining 25% (i.e., 62) of the samples were used for testing. The results of comparing this method with known advanced methods are shown in Table 2 below.

[0192] Table 2

[0193]

[0194] (3) Performance comparison on the FineFS dataset

[0195] For the 2 sub-datasets, which contain 729 and 438 samples respectively, similar to the performance comparison on the RG dataset, 75% of the samples are randomly divided for training, and the remaining 25% for testing. 40% of the training samples are used as labeled samples, and 60% as unlabeled samples. That is: the short program sub-dataset contains 546 training samples and 183 testing samples, among which there are 218 labeled training samples and 328 unlabeled training samples; the free skating sub-dataset contains 328 training samples and 110 testing samples, among which there are 131 labeled training samples and 197 unlabeled training samples. Each sub-dataset of FineFS has two scoring criteria, namely PCS and TES, and the two indicators are calculated respectively. The results of the comparison between this method and the known advanced methods are shown in Table 3 below.

[0196] Table 3

[0197]

[0198] In summary, the present invention reduces the dependence on other pre-trained network modules by quantifying features, simplifies the system structure; improves the interpretability and accuracy of the model by distinguishing the category features and execution situation features of sub-actions; in the first stage, by adjusting the network structure and training parameters, the second-stage training is carried out after ensuring the convergence of the system, improving the training efficiency and stability; generates confidence through similarity and information entropy for pseudo-label screening, avoiding the introduction of noise, and improving the robustness of the system.

Claims

1. A semi-supervised action quality assessment method based on vector quantization and decoupled representation, characterized in that Including: Performing feature extraction on the input video data through a pre-trained primary video feature extraction module to obtain an original feature sequence; Performing a non-linear transformation on the original feature sequence through a pre-trained first multi-layer perceptron to obtain a feature sequence corresponding to the sub-action category; Performing a non-linear transformation on the original feature sequence through a pre-trained second multi-layer perceptron to obtain a feature sequence corresponding to the sub-action execution situation; Performing discrete vector quantization on the feature sequence corresponding to the sub-action category through a pre-trained discrete vector quantization module to obtain a first vector quantization result; Performing continuous vector quantization on the feature sequence corresponding to the sub-action execution situation through a pre-trained continuous vector quantization module to obtain a second vector quantization result; Concatenating the first vector quantization result and the second vector quantization result along the feature dimension and using them as the input of a pre-trained action quality prediction module to obtain an action quality evaluation result.

2. The method according to claim 1, wherein The specific method for performing feature extraction on the input video data through a pre-trained primary video feature extraction module includes: Dividing the input video data into segments with every 16 frames as a slice, performing feature extraction on each slice through a pre-trained primary video feature extraction module, and then obtaining an original feature sequence formed by the features corresponding to all slices; wherein, when the last slice has less than 16 frames, it is padded with a zero tensor.

3. The method according to claim 1, characterized in that, The dimension of the feature sequence corresponding to the sub-action category, the dimension of the feature sequence corresponding to the sub-action execution situation, and the dimension of the original feature sequence are all the same.

4. The method according to claim 1, wherein The specific method for performing discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain a first vector quantization result includes: Randomly initialize a set containing N E embeddings Calculate respectively the distances between each embedding in and the feature sequences corresponding to the sub-action categories, and select the embedding with the smallest distance as the first vector quantization result.

5. The method according to claim 1, wherein The specific method for performing continuous vector quantization on the feature sequence corresponding to the sub-action execution situation to obtain a second vector quantization result is: Randomly initialize a set containing N E embeddings Obtained by calculating the cosine similarity The weights of each embedding in and normalize them, and its expression is: where w j is the normalized weight of the j-th embedding e j in ; e is the feature sequence corresponding to the sub-action execution; k denotes the k-th embedding; 〈.〉represents calculating the cosine similarity; Multiply each embedding in by its corresponding normalized weight and sum them up to obtain the second vector quantization result, and its expression is: Among them is the second vector quantization result 6. The method according to claim 1, characterized in that The pre-trained action quality prediction module includes a network structure with two layers of Transformer-Encoder and a third multi-layer perceptron; The network structure with two layers of Transformer-Encoder is used to globally aggregate the feature data input to the pre-trained action quality prediction module, and take the first element of the output sequence after global aggregation as the input of the multi-layer perceptron; The third multi-layer perceptron is used to perform action quality evaluation based on the input data to obtain an action quality evaluation result.

7. The method according to claim 1, characterized in that, Also including: Calculating the confidence based on the feature sequence corresponding to the sub-action category and the first vector quantization result to obtain a first confidence, and its expression is: where ξ D is the first confidence level; is the first vector quantization result; is the feature sequence corresponding to the sub-action category; <.> represents calculating the cosine similarity; max(.) represents the maximum value function; Randomly initialize a set containing N E embeddings Based on the feature sequence and set corresponding to the sub-action execution situation Calculate the confidence level to obtain the second confidence level, and its expression is: where w j is the normalized weight corresponding to the j-th embedding e j in; is the feature sequence corresponding to the execution of the sub-action; e k represents the k-th embedding; <.> represents calculating the cosine similarity; ξ C is the second confidence; N E is the total number of randomly initialized embeddings; log(.) represents the logarithm to the base 2; Multiplying the first confidence and the second confidence element by element and taking the average to obtain the confidence of the action quality evaluation result of the current input video data, and its expression is: where is the confidence of the action quality assessment result of the currently input video data; M is the sequence length of the first confidence and the second confidence; is the i-th element in the first confidence; is the i-th element in the second confidence.

8. A system based on the method according to any one of claims 1 to 7, characterized in that, Including a pre-trained primary video feature extraction module, a pre-trained first multi-layer perceptron, a pre-trained second multi-layer perceptron, a pre-trained discrete vector quantization module, a pre-trained continuous vector quantization module, and a pre-trained action quality prediction module; wherein: The pre-trained primary video feature extraction module is used to perform feature extraction on the input video data to obtain an original feature sequence; A pre-trained first multi-layer perceptron for performing a non-linear transformation on the original feature sequence to obtain a feature sequence corresponding to the sub-action category; A pre-trained second multi-layer perceptron for performing a non-linear transformation on the original feature sequence to obtain a feature sequence corresponding to the sub-action execution situation; A pre-trained discrete vector quantization module for performing discrete vector quantization on the feature sequence corresponding to the sub-action category to obtain a first vector quantization result; A pre-trained continuous vector quantization module for performing continuous vector quantization on the feature sequence corresponding to the sub-action execution situation to obtain a second vector quantization result; A pre-trained action quality prediction module for concatenating the first vector quantization result and the second vector quantization result along the feature dimension, and performing action quality evaluation based on the concatenated result to obtain an action quality evaluation result.

9. The system according to claim 8, wherein The pre-training method of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module and the action quality prediction module includes two stages. Among them, the first stage includes the following operations: Perform the training preparation of the first stage, including: Divide the labeled training set, unlabeled training set, and validation set; initialize the system hyperparameters, number of training epochs, learning rate, optimizer, and early stopping conditions; Perform the iterative training of the first stage, including: Take samples from the labeled training set in batches and input them into the system, obtain the total system loss value and perform gradient backpropagation, and update the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module and the action quality prediction module through the optimizer; Take samples from the unlabeled training set in batches and input them into the system, obtain the total system loss value and perform backpropagation, and update the parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron and the discrete vector quantization module through the optimizer; Take samples from the validation set in batches and input them into the system, calculate the performance index of the system after obtaining the action quality prediction results of all validation samples and the true action quality; When the first stage reaches the maximum number of training epochs, or the performance index of the system obtained from the validation set meets the early stopping condition, stop the iterative training of the first stage and enter the second stage; the second stage includes the following operations: Perform the training preparation of the second stage, including: Set the number of training epochs, learning rate, optimizer, pseudo-label screening confidence threshold, and early stopping conditions; Perform the iterative training of the second stage, including: Take samples from the labeled training set in batches and input them into the system, obtain the total system loss value and perform gradient backpropagation, and update the parameters of the primary video feature extraction module, the first multi-layer perceptron, the second multi-layer perceptron, the discrete vector quantization module, the continuous vector quantization module and the action quality prediction module through the optimizer; Take samples from the unlabeled training set in batches and input them into the system, obtain the corresponding action quality prediction results and confidence levels; add the unlabeled training samples corresponding to the confidence levels higher than the pseudo-label screening confidence threshold to the labeled training set; Samples are taken from the validation set in batches and input into the system. After obtaining the action quality prediction results of all validation samples, the performance metrics of the system are calculated in comparison with the true action quality. When the maximum number of training epochs in the second stage is reached, or when the performance metrics of the system obtained from the validation set meet the early stopping criteria, the iterative training in the second stage is stopped, and the pre-training of the system is completed.

10. The system according to claim 9, wherein The calculation method of the total system loss value includes: Calculating the triplet loss based on the feature sequence corresponding to the sub-action category output by the first multi-layer perceptron and the feature sequence corresponding to the sub-action execution situation output by the second multi-layer perceptron. Its expression is: Among them, D + and D - are both intermediate parameters; is the j-th element in the feature sequence corresponding to the sub-action category; is the k-th element in the feature sequence corresponding to the sub-action category; dist(.) represents the dist function; max j≠k (.) represents taking the maximum value; and are the j-th element and the k-th element in the feature sequence corresponding to the sub-action execution situation respectively; min j,k (.) represents taking the minimum value; α tri is a hyperparameter used to control loss calculation; is the triplet loss; Calculating the discrete vector quantization loss. Its expression is: Among them is the discrete vector quantization loss; is the feature sequence corresponding to the sub-action category; is the first vector quantization result; sg(.) represents the gradient truncation operation; β is the proximity rate control hyperparameter; Randomly initialize a set containing N E embeddings Calculate the continuous vector quantization loss, and its expression is: Among them means to take out single embeddings one by one from the set and take the embedding corresponding to the smallest distance based on the L2 distance with , that is is the feature sequence corresponding to the execution situation of the sub-action; e j is the j-th embedding in is the continuous vector quantization loss Calculating the prediction loss. Its expression is: Wherein is the prediction loss; is the action quality evaluation result of the currently input video data; y is the true action quality result of the currently input video data; Calculating the total loss: Among them is the pre-trained as above; λ1, λ2, λ3, and λ4 are weight hyperparameters; when the currently input video sample is an unlabeled sample, λ1 = 0.

Images