Self-distillation action quality evaluation method based on location awareness and reward feedback

Abstract

The application discloses a self-distillation action quality evaluation method based on position awareness and reward feedback, which firstly acquires an action quality evaluation data set, obtains space-time features through a multi-scale space-time feature extraction backbone network based on a deep three-dimensional convolution; secondly, the space-time features are respectively input into corresponding time-channel attention modules to obtain enhanced space-time features; the enhanced space-time features are reconstructed to a standard feature dimension space through a feature reconstruction module, and adaptive weights are obtained by an adaptive weight generator; finally, the reconstructed space-time features are respectively input into an action category classifier and a shared regression head, and a loss is calculated by combining the adaptive weights for training; in the inference stage, the action category classifier and the adaptive weight generator are removed to obtain an action quality evaluation result. The application fuses bottom-level detail features and high-level abstract representations, guarantees the precision, improves the operation efficiency and enhances the overall knowledge distillation effect.

Description

Technical Field

[0001] This invention relates to the field of motion quality assessment, and more particularly to a self-distillation motion quality assessment method based on position perception and reward feedback. Background Technology

[0002] Action quality assessment (AQA) aims to automatically evaluate the quality of an individual's performance when executing a sequence of actions using computers, and has received increasing attention in the field of computer vision in recent years. Currently, AQA research is mainly applied in scenarios such as healthcare, sports, and skills assessment. By quantitatively analyzing the accuracy and standardization of actions, it provides a scientific basis for personalized training guidance, rehabilitation effect monitoring, and professional skills certification, possessing significant practical value and social significance. However, efficiently capturing fine-grained intra-class variations and identifying subtle differences in actions within the same scenario and action type remains a challenge.

[0003] In motion quality assessment research, most existing work has focused on improving the prediction accuracy of models. Stacking multiple complex modules has improved the accuracy of motion quality assessment. However, excessive model complexity significantly increases computational overhead, and in scenarios with limited hardware resources (such as mobile devices or edge terminals), it can easily lead to problems such as inference latency and memory overflow, severely limiting the scalability and large-scale deployment capability of models in practical applications. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a self-distillation motion quality assessment method based on position awareness and reward feedback. This method, building upon a lightweight model, introduces position awareness and reward feedback, enabling student modules at different levels to adaptively learn knowledge from the teacher module. Simultaneously, the method provides higher rewards to student modules that are closer to the teacher module, thereby guiding lower-performing student modules to learn more effectively. This helps improve the overall knowledge distillation effect, achieving a precise mapping from motion to score and effectively enhancing the accuracy of motion quality assessment.

[0005] The technical problem solved by this invention includes the following steps:

[0006] S1. Obtain the motion quality assessment dataset and divide it into training, validation, and test data.

[0007] S2. Construct a multi-scale spatiotemporal feature extraction backbone network based on deep 3D convolution (e.g., 3D convolutional neural network architectures such as C3D and I3D), and divide the backbone network into k cascaded feature extraction sub-modules Block1-Block2. k This is done by extracting spatiotemporal features layer by layer. Specifically, Block1 to Block...k-1 As a student module, Block k As a teacher module, it inputs continuous motion image frames from the dataset into a feature extraction backbone network. The network then extracts spatiotemporal features layer by layer through k feature extraction sub-modules, yielding spatiotemporal features f1, f2, ..., f... i ,…,f k , Where f i For Block i The extracted spatiotemporal features are: C represents the number of input channels, T represents the number of image frames in the input video, and H and W represent the height and width of the input image frame, respectively.

[0008] S3. Calculate the spatiotemporal features f1, f2, ..., f from the output of step S2. i ,…,f k The inputs are respectively fed into the corresponding time-channel attention modules TCAM1-TCAM. k Temporal attention weights and channel attention weights are generated for each spatiotemporal feature from both the temporal and channel dimensions to adaptively enhance key information. The attention weights are then multiplied by the spatiotemporal features to obtain the enhanced spatiotemporal features. in spatiotemporal features f i Time Elapsed – Channel Attention Module (TCAM) i The obtained enhanced spatiotemporal features. The specific steps of the time-channel attention module are as follows:

[0009] S31. For each spatiotemporal feature f i The intermediate spatiotemporal features were obtained by applying average pooling and max pooling operations respectively. and Among them, the intermediate spatiotemporal feature f avg It focuses on reflecting global average information that reflects spatiotemporal characteristics, and intermediate spatiotemporal characteristics f max Highlighting and preserving local salient information in spatiotemporal features.

[0010] S32. The intermediate spatiotemporal features f avg and f max The inputs are fed into a shared 3D convolutional network, which employs a "dimensionality reduction-activation-dimensionality increase" structure. The first convolutional layer reduces the number of channels C to C / / ratio, where ratio is the compression ratio and / / indicates integer division. Then, a ReLU activation function performs a non-linear numerical mapping. Finally, the second convolutional layer restores the original number of channels, generating the average channel weights. and maximum channel weight All of their dimensions are Finally, for and The values ​​are summed and averaged, and the channel attention weights A are obtained using the Sigmoid function. c .

[0011] S33. In the temporal attention branch, first exchange the input spatiotemporal features. The channel and frame sequence dimensions are obtained. Then, following the same calculation method as the channel attention weights in steps S31 and S32, the temporal attention weight A is calculated. t .

[0012] S34. Convert the channel attention weights A output from step S32. c With spatiotemporal characteristics f i Perform dot product weighting, and add the time attention weight A output from step S33. t With spatiotemporal characteristics f i The array is weighted by dot product, and the frame sequence dimension and channel dimension are swapped using a dimension transformation function. The spatiotemporal features weighted by channel attention and temporal attention are then summed to output the enhanced spatiotemporal features.

[0013] S4. The enhanced spatiotemporal features output in step S3 The inputs are respectively fed into the corresponding feature reconstruction modules Reshape1-Reshape k The enhanced spatiotemporal features are reconstructed into a standard feature dimension space to optimize the alignment features and obtain reconstructed spatiotemporal features of a unified dimension. in To enhance spatiotemporal features After feature reconstruction module Reshape i The reconstructed spatiotemporal features are obtained. The specific steps of the feature reconstruction module are as follows:

[0014] S41. Use 3×3×3 convolutional kernels to enhance spatiotemporal features. Convolutional operations are performed, followed by batch normalization, and finally, the ReLU activation function is used to enhance the nonlinear expressive power, thus obtaining intermediate enhanced spatiotemporal features.

[0015] S42. Enhance the intermediate spatiotemporal features output from step S41 using a 1×1×1 convolution kernel. The dimensionality is increased, the increased features are batch normalized, and the ReLU activation function is used for nonlinear mapping to complete the feature reconstruction and optimization.

[0016] S5. Reconstruct the spatiotemporal features output from step S4. The adaptive weight generator is input, and adaptive weights λ = {λ1, λ2, ..., λ3} are obtained by combining position awareness and reward feedback. i ,…,λk-1}, where λ i It is the i-th student module Block i The corresponding adaptive weights. The specific steps of the adaptive weight generator are as follows:

[0017] S51. Calculate the Block of the i-th student module respectively. i Reconstructed spatiotemporal features And Teacher Module Block k Reconstructed spatiotemporal features Similarity between Based on location awareness, calculate the anomaly feature metric d = {d1, d2, ..., d...} i ,…,d k-1}, where d i This represents the i-th student module Block. i The anomaly feature measurement, specifically the formula is as follows:

[0018]

[0019] in This means if r j >r i If the result is positive, then take 1; otherwise, take 0. Indicates taking 0 and The maximum value in.

[0020] S52. Using reward feedback to adjust the abnormal feature metric d i We obtain the adaptive weight λ. By giving higher rewards to student modules that are closer to the teacher's module, we guide the poorly performing student modules to learn more effectively. The specific steps of the reward feedback are as follows:

[0021] S521. Use the Sigmoid function F Sigmoid For similarity r i Normalize to obtain reward values ​​reward1, reward2, ..., reward i ,…,reward k-1 reward i This represents the i-th student module Block. i The corresponding reward value. Finally, the reward value. i Multiplying by the scaling factor μ and the weight η, the result is the anomaly feature measure d. i Anomaly metric for reward feedback obtained by summing This represents the i-th student module Block. i Anomaly measurement of reward feedback.

[0022] S522. Use the Softmax function to measure the abnormal features of reward feedback. After normalization, we obtain the adaptive weights λ = {λ1, λ2, ..., λ} i ,…,λ k-1}, where λ i It is the i-th student module Block i The corresponding adaptive weights.

[0023] S6. Calculate the action category loss: Reconstruct the spatiotemporal features output from step S4. The input is fed into an action category classifier, passing through a Multi-Layer Perceptron (MLP) consisting of three hidden layers and a Softmax layer to generate a predicted category probability distribution. The action category labels are then encoded using one-hot encoding to generate a label category probability distribution. Finally, the adaptive weight λ output from step S5 is used to calculate the cross-entropy loss between the predicted category probability distribution and the label category probability distribution. The specific steps for calculating the action category loss are as follows:

[0024] S61. Reconstructing Spatiotemporal Features The input is fed into an MLP consisting of three hidden layers and a Softmax layer, each of which is a fully connected layer and a ReLU activation function. The predicted class probability distribution is then calculated. in Block represents the i-th feature extraction submodule. i The corresponding predicted category probability distribution.

[0025] S62. Generate a label category probability distribution by one-hot encoding the action category labels. in, Represents the probability distribution of label categories The predicted probability of the i-th action category, where m represents the number of all action categories in the classification task.

[0026] S63. Combining the adaptive weight λ output from step S5, calculate the predicted category probability distributions for the teacher module and student module respectively using the cross-entropy formula. and label category probability distribution The difference between them is used to obtain the action category cross-entropy loss L by adaptive weighted summation. sup-ce .

[0027] S7. Calculate the action score loss: Reconstruct the spatiotemporal features output from step S4. The input is fed into a shared regression head, guiding shallow features to achieve consistency with deep semantics. After passing through a multilayer perceptron (MLP) consisting of a hidden layer and a softmax layer, a predicted score probability distribution is generated, with the score corresponding to the highest predicted probability item. To predict the score, the KL divergence loss between the predicted score probability distribution and the Gaussian distribution is calculated by mapping the score labels to a Gaussian distribution and combining the adaptive weights λ output from step S5. The specific steps for calculating the action score loss are as follows:

[0028] S71. Reconstructing Spatiotemporal Features The input consists of a multilayer perceptron (MLP) with a single hidden layer and a softmax layer, where the hidden layer is composed of a fully connected layer and a ReLU activation function. The predicted score probability distribution is then calculated. in Block represents the i-th feature extraction submodule. i The corresponding predicted score probability distribution.

[0029] S72. Map the score labels s to a Gaussian distribution. The mean is the score label s, and the standard deviation is σ. The specific formula is as follows: Generate a corresponding Gaussian probability distribution F for each score label s. gasi (u1),F gasi (u2),…,F gasi (u i ),…,F gasi (u n The mean is the score label s, the standard deviation is σ, and F gasi (u i ) is a Gaussian function.

[0030] The distribution F of the score labels s gasi (u1),F gasi (u2),…,F gasi (u i ),…,F gasi (u n Normalization is performed to obtain the final label score probability distribution.

[0031] S73. Combining the adaptive weight λ output from step S5, calculate the predicted score probability distributions for the teacher module and the student module respectively using the KL divergence formula. and label score probability distribution The difference between them is used to obtain the action score KL loss L by adaptive weighted summation. sup-kl .

[0032] S8. Calculate the self-distillation loss: The self-distillation loss of this model is composed of the L2 norm loss of the teacher and the student. stu-l2 The KL divergence loss between teachers and students L stu-kl It is composed of various components. The specific steps for calculating self-distillation loss are as follows:

[0033] S81. Reconstruct the spatiotemporal features output from step S4. The L2 norm loss of the teacher module and each student module is calculated by inputting the values ​​into the L2 norm formula. Then, the L2 norm loss L of the teacher module and student modules is obtained by weighted summation using adaptive weights λ. stu-l2 .

[0034] S82. Calculate the probability distribution of the predicted scores output in step S71. The KL divergence loss is calculated by inputting the data into the KL divergence formula and then combining it with the adaptive weight λ to calculate the KL divergence loss L between the teacher module and the student modules. stu-kl .

[0035] S9. During the model training phase, calculate the total model loss on the training set. The total model loss is the action class cross-entropy loss L output in step S63. sup-ce Step S74 outputs the action score KL divergence loss L sup-kl The L2 norm loss L of the teacher module and student module output in step S81 stu-l2 The KL divergence loss L for teachers and students output in step S82. stu-kl It is composed of these components, and the Adam model parameters are iteratively optimized through adaptive moment estimation.

[0036] S10. During the inference phase of the model, the action category classifier and adaptive weight generator are removed, and only the feature extraction backbone network, temporal-channel attention module, feature reconstruction module, and shared regression head are used to obtain the final prediction score. That is, the results of the motion quality assessment.

[0037] The beneficial effects of this invention are as follows: The model adopts a hierarchical feature extraction architecture, progressively integrating low-level detailed features with high-level abstract representations through a cross-level spatial-semantic fusion mechanism, thereby improving the anti-interference ability and generalization performance of feature representation. During inference, the optimal feature level sub-model is dynamically selected based on task complexity, ensuring both accuracy and significantly improving operational efficiency; the temporal-channel attention module guides the model to focus on key information, reducing interference from redundant information; adaptive weights are generated based on position awareness and combined with reward feedback for loss function calculation, effectively guiding the poorly performing student module to learn and reasonably adjusting the loss calculation during training to improve model performance; a feature reconstruction module is used to uniformly map multi-scale features and transform them into a standard dimensional space to optimize feature alignment; an action category classifier module is added, allowing the model to learn professional knowledge in the action domain to improve the accuracy of model prediction; the use of a shared regression head not only reduces model storage overhead but also guides shallow features and deep semantics to achieve consistency, that is, shallow features are guided to acquire deep semantic information, thereby accelerating the convergence and alignment of shallow features to deep semantic representations and promoting the semantic consistency of multi-level features. Attached Figure Description

[0038] Figure 1 This is an architecture diagram of a self-distillation action quality assessment model based on location awareness and reward feedback.

[0039] Figure 2 Here is a diagram of the time-channel attention module structure;

[0040] Figure 3 This is a structural diagram of the feature reconstruction module. Detailed Implementation

[0041] The following attached diagrams further illustrate the self-distillation action quality assessment method based on location awareness and reward feedback, such as... Figure 1 As shown, it includes the following steps:

[0042] S1. Use the FineDiving (a fine-grained diving dataset containing 3000 diving videos) and MTL-AQA (a multi-task learning action quality assessment dataset containing 1412 diving videos) action quality assessment datasets as training, validation, and testing data.

[0043] S2. An improved feature extraction backbone network based on the I3D structure is constructed, dividing the backbone network into five cascaded feature extraction sub-modules (Blocks 1-5) to extract spatiotemporal features layer by layer, enhancing representational capabilities while maintaining computational efficiency. Blocks 1 to 4 serve as student modules, and Block 5 as the teacher module. Continuous action image frames are input into the feature extraction backbone network, and spatiotemporal features are extracted layer by layer through the five sub-modules, yielding spatiotemporal features f1, f2, f3, f4, and f5. Where f i For Block i The extracted spatiotemporal features are defined as follows: C represents the number of input channels, T represents the number of frames in the input video, and H and W represent the height and width of the input video, respectively. The specific design of the feature extraction backbone network is as follows:

[0044] S21. Input continuous motion image frames Where C represents the number of input channels, T represents the number of frames in the input video, and H and W represent the height and width of the input video, respectively. First, the feature extraction submodule Block1 extracts the continuous motion image frames X from the input through a 7×7×7 three-dimensional convolution operation. img The initial spatiotemporal features are obtained, and then downsampling is performed through a 1×3×3 max pooling operation to suppress redundant features and achieve controllable compression of the feature space dimension. Finally, the extracted spatiotemporal features f1 are output. The specific formula is as follows:

[0045]

[0046] in, This indicates that max pooling is performed using a 1×3×3 three-dimensional pooling window. This indicates that a 7×7×7 three-dimensional convolution kernel is used for the convolution operation.

[0047] S22. Input the spatiotemporal feature f1 output in step S21, and first perform a 1×1×1 convolution operation. Channel dimension compression is performed to optimize parameter efficiency, followed by a 3×3×3 convolution operation. Local spatiotemporal features are captured, and finally, a 1×3×3 max pooling operation is performed. The features are further compressed to output the extracted spatiotemporal features f2, as shown in the following formula:

[0048]

[0049] in, This indicates that max pooling is performed using a 1×3×3 three-dimensional pooling window. This indicates that a 3×3×3 three-dimensional convolution kernel is used for the convolution operation. This indicates that a 1×1×1 three-dimensional convolution kernel is used for the convolution operation.

[0050] S23. Input the spatiotemporal features f2 output in step S22, and fuse the spatiotemporal features of different receptive fields by cascading two Inception modules and using parallel multi-scale convolution operations. Finally, use 3×3×3 max pooling. To achieve dimensionality reduction of spatiotemporal features, the extracted spatiotemporal features f3 are output. The specific formula is as follows:

[0051]

[0052]

[0053]

[0054]

[0055]

[0056] in, This indicates that a 1×1×1 three-dimensional convolution kernel is used for the convolution operation. This indicates that a 3×3×3 three-dimensional convolution kernel is used for the convolution operation. This indicates that max pooling is performed using a 3×3×3 three-dimensional pooling window, F Concat The function () represents concatenating the input features along the channel dimension. Icp 2 () indicates that the process has been processed by two cascaded Inception modules.

[0057] S24. Input the spatiotemporal features f3 output in step S23, and construct a five-layer progressive Inception cascaded topology F. Icp 5 (), through a hierarchical nonlinear mapping mechanism, the semantic representation is abstracted layer by layer, and finally through 2×2×2 max pooling. Extract high-dimensional abstract features and output the extracted spatiotemporal features f4, as shown in the following formula:

[0058]

[0059] in, This indicates that max pooling is performed using a 2×2×2 three-dimensional pooling window, F Icp 5 () indicates that the process has been processed through 5 cascaded Inception modules.

[0060] S25. Input the spatiotemporal feature f4 output in step S24, process it in parallel with convolutional kernels of different sizes through two Inception modules, and use 2×7×7 max pooling operation. High-dimensional abstract features are extracted, and finally the extracted spatiotemporal features f5 are output. The specific formula is as follows:

[0061]

[0062] in, This indicates that max pooling is performed using a 2×7×7 three-dimensional pooling window, F Icp 2 () indicates that the process has been processed by two cascaded Inception modules.

[0063] The high-dimensional latent spatiotemporal feature f5 generated by this module serves as the knowledge carrier for the teacher module. Through feature distillation and response distillation transfer paths, it achieves multi-level knowledge transfer, transmitting the contained knowledge information to the shallow student module.

[0064] S3. Input the spatiotemporal features f1, f2, f3, f4, and f5 output from step S2 into the corresponding time-channel attention modules TCAM1-TCAM5, respectively. Figure 2 As shown, temporal attention weights and channel attention weights are generated for each spatiotemporal feature from both the temporal and channel dimensions to adaptively enhance key information. The attention weights are then multiplied by the spatiotemporal features to obtain the enhanced spatiotemporal features. in spatiotemporal features f i Time Elapsed – Channel Attention Module (TCAM) i The enhanced spatiotemporal features obtained after processing. The specific steps of the time-channel attention module are as follows:

[0065] S31. For each spatiotemporal feature f i Average pooling F was used respectively AvgPool () and maximum pooling F MaxPool The () operation yields intermediate spatiotemporal features. and Among them, the intermediate spatiotemporal feature f avg It focuses on reflecting global average information that reflects spatiotemporal characteristics, and intermediate spatiotemporal characteristics f max This approach emphasizes preserving local salient information within spatiotemporal features. The specific formula is as follows:

[0066] f avg =F AvgPool (f i )

[0067] f max =F MaxPool (f i )

[0068] S32. The intermediate spatiotemporal features f avg and f max The inputs are respectively fed into a shared 3D convolutional network, which adopts a "dimensionality reduction-activation-dimensionality increase" structure design, where the first convolutional layer F... conv1 () Reduce the number of channels C to C / / ratio, where ratio is the compression ratio and / / means integer division. Then, pass it through the ReLU activation function F. ReLU () performs nonlinear numerical mapping, and finally passes through a second convolutional layer F conv2 () Restore to the original number of channels and generate the average channel weight. and maximum channel weight All of their dimensions are Finally, for and The values ​​are summed and averaged, and the channel attention weights A are obtained using the Sigmoid function. c The specific formula is as follows:

[0069]

[0070]

[0071]

[0072] S33. In the temporal attention branch, first exchange the input spatiotemporal features. The channel and frame sequence dimensions are obtained. Then, following the same calculation method as the channel attention weights in steps S31 and S32, the temporal attention weight A is calculated. t That is: for the spatiotemporal features f after dimensional transformation i First, the intermediate spatiotemporal features are obtained by average pooling and max pooling, respectively. and Then, the intermediate spatiotemporal features are input into a shared 3D convolutional network, and after passing through the convolutional layer F... conv1 (), ReLU activation function F ReLU () and convolutional layer F conv2 () to obtain the average time weight and maximum time weight Finally and The sums are averaged and then input into the Sigmoid function F. Sigmoid (), to obtain the time attention weight A t The specific formula is as follows:

[0073] f i ′=Ftranspose (f i ,ε)

[0074] f′ avg =F AvgPool (f i ′)

[0075] f′ max =F MaxPool (f i ′)

[0076]

[0077]

[0078]

[0079] Among them, F transpose The () function rearranges the dimensions of the features according to the new dimension order specified by the feature dimension parameter ε, where the dimension order specified by ε is (1,0,2,3).

[0080] S34. Convert the channel attention weights A output from step S32. c With spatiotemporal characteristics f i Perform dot product weighting, and add the time attention weight A output from step S33. t With spatiotemporal characteristics f i Perform dot product weighting and then pass it through the dimension transformation function F. transpose () Swap the frame sequence dimension and channel dimension, add the spatiotemporal features that have undergone channel attention weighting and temporal attention weighting, and output the enhanced spatiotemporal features. The specific formula is as follows:

[0081]

[0082] S4. The enhanced spatiotemporal features output in step S3 The inputs are respectively fed into the corresponding feature reconstruction modules Reshape1-Reshape5, such as... Figure 3 As shown, the enhanced spatiotemporal features are reconstructed into a standard feature dimension space to optimize the alignment features, resulting in reconstructed spatiotemporal features of a unified dimension. in To enhance spatiotemporal features After feature reconstruction module Reshape i The reconstructed spatiotemporal features are obtained. The specific steps of the feature reconstruction module are as follows:

[0083] S41. Use 3×3×3 convolutional kernels to enhance spatiotemporal features. Convolutional operations are performed, followed by batch normalization to improve the model's stability and convergence. Finally, the ReLU activation function is used to enhance nonlinear expressiveness and improve the model's generalization ability, thus obtaining intermediate enhanced spatiotemporal features.

[0084] S42. Enhance the intermediate spatiotemporal features output from step S41 using a 1×1×1 convolution kernel without changing the temporal and spatial dimensions. The dimensionality is increased, the increased features are batch normalized, and the ReLU activation function is used for nonlinear mapping to complete the feature reconstruction and optimization.

[0085] S5. Reconstruct the spatiotemporal features output from step S4. The adaptive weight generator is input, and adaptive weights λ = {λ1, λ2, λ3, λ4} are obtained by combining position awareness and reward feedback, where λ i It is the i-th student module Block i The corresponding adaptive weights. The specific steps of the adaptive weight generator are as follows:

[0086] S51. Calculate the Block of the i-th student module respectively. i Reconstructed spatiotemporal features Spatiotemporal characteristics of the reconstruction of the teacher module Block 5 Similarity between The similarity is obtained as r = {r1, r2, r3, r4}, where r i This represents the i-th student module Block. i The similarity between the reconstructed spatiotemporal features of the student module and the reconstructed spatiotemporal features of the teacher module (Block 5) is calculated. Since the deepest sub-module of the backbone network is used as the teacher module for feature extraction, the reconstructed spatiotemporal features generated by the deeper student modules are similar. Reconstructed spatiotemporal features generated by the teacher module The similarity should be higher than that of the shallow student module. and The similarity is higher, and the anomaly feature metric d = {d1, d2, d3, d4} is calculated based on location awareness, where d i This represents the i-th student module Block. i The anomaly feature measurement, specifically the formula is as follows:

[0087]

[0088] in This means if r j >r i If the result is positive, then take 1; otherwise, take 0. Indicates taking 0 and The larger value in the range.

[0089] S52. However, since d1 is the anomaly metric of the first student module Block1, the calculated d1 is always 0. Therefore, reward feedback is used to adjust the anomaly metric d. i By awarding higher rewards to student modules that are closer to the teacher's module, students with lower-performing modules are guided to learn more effectively. The specific steps for reward feedback are as follows:

[0090] S521. Use the Sigmoid function F Sigmoid For similarity r i After normalization, the closer the result is to 1, the more similar it is to the teacher's characteristics, and the more it should be rewarded, resulting in reward values ​​reward1, reward2, reward3, and reward4, where reward... i This represents the i-th student module Block. i The corresponding reward value is as follows: the closer the reward value is to 0, the better the student has learned the module. Finally, the reward value is... i Multiplying by the scaling factor μ and the weight η, the result is the anomaly feature measure d. i Anomaly metric for reward feedback obtained by summing This represents the i-th student module Block. i The abnormal feature measurement of reward feedback. The specific formula is as follows:

[0091] reward i =1-F Sigmoid (r i )

[0092] μ = mean(d)

[0093]

[0094] Where μ is the scaling factor, and its value is for all d i The average value is used to adjust the reward value. i The dimensions of the anomalous feature measure d are made to match those of the anomalous feature measure d. i Being on the same order of magnitude, η is an adjustment parameter used to adjust the weight of the reward value.

[0095] S522. Use the Softmax function to measure the abnormal features of reward feedback. After normalization, we obtain the adaptive weights λ = {λ1, λ2, λ3, λ4}, where λ i It is the i-th student module Block i The corresponding adaptive weights.

[0096] S6. Calculate the action category loss: Reconstruct the spatiotemporal features output from step S4. The input is fed into an action category classifier, which passes through a Multilayer Perceptron (MLP) consisting of three hidden layers and a Softmax layer to generate a predicted category probability distribution. The action category labels are then encoded using one-hot encoding to generate a label category probability distribution. Finally, the adaptive weights λ output from step S5 are combined to calculate the cross-entropy loss between the predicted category probability distribution and the label category probability distribution. The specific steps for calculating the action category loss are as follows:

[0097] S61. Reconstructing Spatiotemporal Features The input is fed into an MLP consisting of three hidden layers and a Softmax layer F. Softmax Each hidden layer consists of a fully connected layer. and ReLU activation function F ReLU It consists of (). Among them, Let represent the fully connected layer in the i-th hidden layer. The predicted class probability distribution is calculated. in Block represents the i-th feature extraction submodule. i The corresponding predicted category probability distribution. The specific formula is as follows:

[0098]

[0099] Indicates the output layer, The features of the layer are mapped to the probability distribution of the action category.

[0100] S62. Generate a label category probability distribution by one-hot encoding the action category labels. in, Represents the probability distribution of label categories Let m represent the predicted probability of the i-th action category in the classification task, and m represent the total number of action categories. After one-hot encoding, the probability of the category corresponding to the true label is 1, and the probability of the other categories is 0.

[0101] S63. Combining the adaptive weights λ output from step S5, calculate the predicted class probability distribution using the cross-entropy formula. and label category probability distribution The difference between them yields the action category cross-entropy loss L. sup-ce The specific formula is as follows:

[0102]

[0103]

[0104] in, It is the predicted probability of the j-th action category in the probability distribution of the i-th predicted category. Represents the probability distribution of label categories The predicted probability of the j-th action category is given by the adjustment factor ρ, which adjusts the contribution of the teacher module's loss to the total loss to avoid weakening its influence due to the weighted average of other loss items, thereby ensuring the effectiveness of knowledge transfer. m represents the number of all action categories in the classification task.

[0105] S7. Calculate the action score loss: Reconstruct the spatiotemporal features output from step S4. The input is fed into a shared regression head, guiding shallow features to achieve consistency with deep semantics. After passing through a multilayer perceptron (MLP) consisting of a hidden layer and a softmax layer, a predicted score probability distribution is generated, with the score corresponding to the highest predicted probability item. To predict the score, the KL divergence loss between the predicted score probability distribution and the Gaussian distribution is calculated by mapping the score labels to a Gaussian distribution and combining the adaptive weights λ output from step S5. The specific steps for calculating the action score loss are as follows:

[0106] S71. Reconstructing Spatiotemporal Features The input consists of a multilayer perceptron (MLP) with one hidden layer and a softmax layer (F). Softmax (), this hidden layer is composed of a fully connected layer and ReLU activation function F ReLU The predicted score probability distribution is calculated using the parentheses (). in Block represents the i-th feature extraction submodule. i The corresponding probability distribution of predicted scores. The specific formula is as follows:

[0107]

[0108] in, Indicates the output layer, The feature maps of the layer to the number of score intervals n.

[0109] S72. Map the score labels s to a Gaussian distribution. The mean is the score label s, and the standard deviation is σ. The specific formula is as follows: Generate a corresponding Gaussian probability distribution F for each score label s. gasi (u1),F gasi (u2),…,F gasi (u i ),…,F gasi (u n The mean is the score label s, the standard deviation is σ, and F gasi (u i Let be the Gaussian function, and its formula is:

[0110]

[0111] Among them, u={u1,u2,...,u n} represents uniform discrete sampling over a continuous Gaussian distribution interval, where each u i It corresponds to a specific location on the Gaussian distribution and is used to approximate a continuous probability distribution.

[0112] S73. Distribution F of the score labels s output in step S72 gasi (u1),F gasi (u2),…,F gasi (u i ),…,F gasi (u n Normalization is performed to obtain the final label score probability distribution.

[0113] S74. Combining the adaptive weight λ output from step S5, calculate the predicted score probability distribution using the KL divergence formula. and label score probability distribution The difference between them yields the action score KL loss L. sup-kl The specific formula is as follows:

[0114]

[0115]

[0116] Wherein, the KL divergence formula L kl In (p||q), p represents the label probability distribution, q represents the predicted probability distribution, n represents the number of score intervals, j represents the index of the j-th action category, and the action score KL loss is L. sup-kl In the formula, Block represents the i-th feature extraction submodule. i The corresponding predicted score probability distribution, Let represent the probability distribution of label scores, and let ρ be the adjustment factor that adjusts the contribution of the teacher module's loss to the total loss. λ i This represents the i-th student module Block. i The corresponding adaptive weights.

[0117] S8. Calculate the self-distillation loss: The self-distillation loss of this model is composed of the L2 norm loss of the teacher and the student. stu-l2 The KL divergence loss between teachers and students L stu-kl It is composed of various components. The specific steps for calculating self-distillation loss are as follows:

[0118] S81. Reconstruct the spatiotemporal features output from step S4. The L2 norm loss L for teachers and students is calculated by inputting the data into the L2 norm formula and combining it with the adaptive weight λ output from step S5. stu-l2 The specific formula is as follows:

[0119]

[0120]

[0121] in, This represents the spatiotemporal characteristics of the reconstruction of the teacher module Block 5. This represents the i-th student module Block. i The reconstructed spatiotemporal features, λ i This represents the i-th student module Block. i The corresponding adaptive weights.

[0122] S82. Calculate the probability distribution of the predicted scores output in step S71. The KL divergence loss L for teachers and students is calculated by inputting it into the KL divergence formula and combining it with the adaptive weight λ output from step S5. stu-kl The specific formula is as follows:

[0123]

[0124] in, This represents the probability distribution of predicted scores for the teacher module Block 5. This represents the i-th student module Block. i The predicted score probability distribution, λ i This represents the i-th student module Block. i The corresponding adaptive weights.

[0125] S9. During the model training phase, calculate the total model loss on the training set. The total model loss is the action class cross-entropy loss L output in step S63. sup-ce Step S74 outputs the action score KL divergence loss L sup-kl The L2 norm loss L for teachers and students output in step S81. stu-l2 The KL divergence loss L for teachers and students output in step S82. stu-kl This is achieved by combining the above methods and iteratively optimizing the model parameters using the Adam method of adaptive moment estimation. The specific formula is as follows:

[0126] L final =L sup-kl +L sup-ce +α·L stu-kl +β·L stu-l2

[0127] Among them, L finalThis represents the total loss of the model, and α and β are hyperparameters used to balance the effects of the two self-distillation losses on the overall loss function.

[0128] S10. During the inference phase of the model, the action category classifier and adaptive weight generator are removed, and only the feature extraction backbone network, temporal-channel attention module, feature reconstruction module, and shared regression head are used to obtain the final prediction score. First, use the maximum index function F argmax () Obtain the index value of the maximum predicted probability Based on the index value of the maximum predicted probability Obtain the final predicted score The specific formula is as follows:

[0129]

[0130]

[0131] Here, argmax represents selecting the index of the maximum value in the vector, and then the predicted score corresponding to the index can be obtained through the maximum value index. Block represents the i-th feature extraction submodule. i The predicted probability of the j-th category in the corresponding predicted score probability distribution, where n represents the number of score intervals. Indicates the first The class score category label, i.e., the predicted score. This will be used as the result of the motion quality assessment.

[0132] Example:

[0133] The steps in this embodiment are the same as those in the specific implementation method, and will not be repeated here. The implementation process and results are shown below.

[0134] To evaluate the lightweight performance of the model, this technique was used to construct a comparative study of the feature extraction submodule on the MTL-AQA benchmark dataset. A multi-dimensional evaluation system was employed, including floating-point operations per second (FLOPs), parameter count, inference time, Spearman Rank Correlation Coefficient (SRCC), and Euclidean norm (R²). l2The Euclidean norm was used as an evaluation metric to explore the optimization effect of self-distillation technology on model compression. The experiment utilized an NVIDIA RTX 3090 GPU hardware platform. The Thop toolkit was used to accurately quantify the computational load of forward propagation, and millisecond-level inference latency was measured using PyTorch's CUDA event interface. To eliminate hardware initialization bias, a mean sampling strategy was adopted, performing 100 warm-up inference tasks followed by 100 effective inferences. For data presentation, Table 1 compares the performance differences of the five-level progressive regression head architecture (Block 1 to Block 5).

[0135] As shown in Table 1 of the experimental results, as the regression head structure expands from Block 1 to Block 5, both the computational load (FLOPs) and the number of parameters in the model exhibit a significant positive correlation. It is worth noting that the Spearman Rank Correlation Coefficient (SRCC) and the Euclidean Norm (R0) also show significant positive correlations. l2 The evaluation metrics improved synchronously during this process, validating the multi-level knowledge transfer characteristics of the self-distillation mechanism: the shallow architecture ensures real-time processing efficiency with lower computational resource consumption, while the deep architecture improves prediction accuracy by utilizing high-order feature extraction capabilities; the two form a complementary and optimized technical path. This phenomenon aligns with the balance theory between complexity and model performance in deep learning, fully demonstrating the effective role of the hierarchical expansion strategy in promoting knowledge transfer.

[0136] Table 1. Lightweighting effect of the model based on the MTL-AQA dataset

[0137]

[0138]

[0139] Table 1 summarizes the key findings: the SRCC of student modules Blocks 1-4 achieved peak performance of 96.1%, 98.0%, 99.6%, and 100%, respectively, with Block 4 surpassing the benchmark value of teacher module Block 5. In terms of computational efficiency, student module Block 1 reduced floating-point operations (FLOPs) and model parameters by 77.6% and 78.7% compared to teacher module Block 5, respectively, and its inference latency was reduced to 29.0% of that of teacher module Block 5. This hierarchical performance evolution reveals the correlation between model architecture selection and hardware adaptation: in edge device scenarios with limited computing power, Blocks 1 to 3 are recommended for low-latency response; while in cloud server environments with sufficient computing resources, Block 4 is recommended for optimal prediction accuracy.

[0140] Table 2 shows that, on the MTL-AQA dataset, the present invention was compared with several state-of-the-art benchmark methods, such as USDL (uncertainty-aware score assignment learning for action quality assessment), MUSDL+PECoP (efficient continuous pre-training of parameters for action quality assessment), GDLT (Likert scoring and grading decoupling for long-term action assessment), and CoFInAI (enhancing action quality assessment through coarse-to-fine instruction alignment). The present invention achieved an SRCC metric of 0.9476 on the MTL-AQA dataset, demonstrating competitive results compared to other benchmark methods and proving the effectiveness of the proposed method.

[0141] Comparative experimental results show that the present invention, through the joint optimization of time-channel attention mechanism, position awareness and reward feedback mechanism and collaborative self-distillation, can effectively capture fine-grained intra-class changes, thereby achieving accurate mapping from action to score and effectively improving the accuracy of action quality assessment.

[0142] Table 2 Performance comparison of the present invention and benchmark methods on the MTL-AQA dataset

[0143]

Claims

1. A method for self-distillation action quality evaluation based on location awareness and reward feedback, characterized in that, The method comprises the following steps: S1. Obtain an action quality evaluation dataset; S2. Construct a deep three-dimensional convolution-based multi-scale space-time feature extraction backbone network, and perform feature extraction on data samples in the dataset to obtain space-time features; The multi-scale space-time feature extraction backbone network is implemented by constructing a multi-scale space-time feature extraction backbone network based on a deep three-dimensional convolution, dividing the backbone network into serial feature extraction sub-modules - , wherein is a student module, is a teacher module; the continuous action image frames in the data set are input into the extraction backbone network, and features are extracted layer by layer through feature extraction sub-modules to obtain space-time features , , wherein is the extracted space-time features, and represent the number of input channels and the number of image frames, and are the height and width of the input image frames;​​ S3. Input the space-time features into corresponding time-channel attention modules respectively, and adaptively enhance key information to obtain enhanced space-time features; S4. Input the enhanced space-time features into corresponding feature reconstruction modules respectively, and reconstruct the enhanced space-time features to a standard feature dimension space to obtain reconstructed space-time features of a unified dimension; S5. Input the reconstructed space-time features into an adaptive weight generator to obtain adaptive weights; The adaptive weight generator is implemented as follows: S51. Calculate the first one respectively. Student Module Reconstructed spatiotemporal features And Teacher Module Reconstructed spatiotemporal features Similarity between Location-aware computing anomaly feature measurement , For the first Student Module Anomaly feature measurement: ； wherein represents if then 1, otherwise 0, represents the maximum of 0 and ; S52. Adjusting the abnormality feature metric with reward feedback , obtaining adaptive weights , as follows; S521. normalizing the similarity using a Sigmoid function to obtain a reward value ​​​​​​​​​​​​​ S522. Use the Softmax function to measure the abnormal features of reward feedback. Normalization is performed to obtain adaptive weights. , It is the first Student Module Corresponding adaptive weights; S6. Input the reconstructed space-time features into an action category classifier and a shared regression head respectively, calculate cross-entropy loss and KL divergence loss in combination with the adaptive weights, and calculate a self-distillation loss for training; in an inference stage, remove the action category classifier and the adaptive weight generator, and the obtained prediction score is the action quality evaluation result.

2. The method of claim 1, wherein, The time-channel attention module is implemented as follows: S31. To the features respectively using average pooling and max pooling, to obtain intermediate spatio-temporal features and ; S32. Intermediate spatiotemporal features and Each input is fed into a shared 3D convolutional network, where the first convolutional layer will have a certain number of channels. Down to , The compression ratio is... This indicates integer division. Then, it undergoes a non-linear numerical mapping using the ReLU activation function, and finally, it is restored to the original number of channels through a second convolutional layer. Generate average channel weights and maximum channel weight ,right and The values ​​are summed, averaged, and then the channel attention weights are obtained using the Sigmoid function. ; S33. Exchange the input spatio-temporal features of the channel and frame sequence dimension, obtaining Then, the time attention weight is calculated in the same way as the channel attention weight in step S31 and step S32 ; S34. Channel attention weights are applied to the spatial-temporal features with spatial-temporal features point-wise multiplication weighting; time attention weights are applied with spatial-temporal features point-wise multiplication weighting, and a dimension transformation function is applied to exchange the frame sequence dimension and the channel dimension; the spatial-temporal features that are applied with channel attention and time attention are added, and an enhanced spatial-temporal feature is output. 3.The method of claim 2, wherein, The feature reconstruction module is specifically implemented as follows: the enhanced spatio-temporal features are convolved and batch normalized using three-dimensional convolution, and then the intermediate enhanced spatio-temporal features are obtained through a ReLU activation function; and The intermediate enhanced spatio-temporal features are processed by dimensionality increasing, batch normalization and non-linear mapping through an activation function to obtain reconstructed spatio-temporal features .

4. The method of claim 3, wherein, The calculation of the cross-entropy loss is implemented as follows: S61. Reconfigure the spatio-temporal features to an action class classifier composed of a multi-layer perceptron (MLP) and a Softmax layer to obtain a predicted class probability distribution , wherein denotes the th feature extraction sub-module corresponding predicted class probability distribution; S62. Generate label class probability distribution by passing the action class label through one-hot encoding , , representing the label class probability distribution the predicted probability of the action class in the th action class, representing the number of all action classes in the classification task; S63. Combining adaptive weights The difference between the predicted class probability distribution of the teacher module and the student module is calculated by the cross-entropy formula respectively and the label class probability distribution The action class cross-entropy loss is obtained by weighted sum according to the adaptive weight .

5. The method of claim 4, wherein, The calculation of the KL divergence loss is implemented as follows: S71. reconstructing spatio-temporal features Input the shared regression head composed of MLP and Softmax layer to obtain the predicted score probability distribution wherein denotes the th feature extraction sub-module corresponding predicted score probability distribution; S72. Map the score labels into a Gaussian distribution with mean score label and standard deviation ; S73. Combined adaptive weights The difference between the predicted score probability distribution of the teacher module and the student module is calculated by the KL divergence formula respectively and the label score probability distribution The action score KL loss is obtained by weighted sum according to the adaptive weights .

6. The method of claim 5, wherein the method further comprises: The calculation of the self-distillation loss is implemented as follows: reconstructing spatio-temporal features into an L2 norm formula to calculate L2 norm loss of the teacher module and each student module, and combine adaptive weights to obtain L2 norm loss of the teacher module and the student module by weighted summation ; The predicted score probability distribution is input into the KL divergence formula to calculate the KL divergence loss of the teacher module and each student module, and combined with the adaptive weight The weighted sum is calculated to obtain the KL divergence loss of the teacher module and the student module ; L2 norm loss of the teacher module and the student module KL divergence loss of the teacher module and the student module Combining to get self-distillation loss.

Images