Multi-person motion quality evaluation based on contrast regression

Abstract

The invention relates to the technical field of video understanding, and discloses a multi-person action quality evaluation method based on contrast regression, which comprises the following steps: S3, introducing an environment adaptability learning module, extracting environment features from video frames, and fusing the environment features and human body posture features to obtain an enhanced feature representation F; according to the multi-person action quality evaluation based on contrast regression, a pre-trained convolutional neural network is introduced through an environment adaptability learning module to extract an environment feature vector of each frame of video, and the environment feature vector is fused with human body posture features to obtain an enhanced feature representation F, and in the training process, a loss function is adjusted to obtain a multi-person action quality evaluation result. Meanwhile, the influence of the evaluation error of the action quality and the environmental characteristics is considered, the robustness and accuracy of the model in a complex environment are improved, for example, in outdoor sports, the introduction of the environmental characteristics can make up for inaccurate detection caused by direct sunlight or shadow areas, and the model performance is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video understanding, and in particular to a multi-person action quality assessment based on contrast regression. Background Art

[0002] Action quality assessment is an important task that aims to evaluate the accuracy and degree of performing a specific action. This task is usually modeled as a score regression problem, and the quality score of the action is predicted by a machine learning model. Although existing action quality assessment (AQA) models have achieved remarkable results in sports, for example, at the 2020 Tokyo Olympics, the AI ​​scoring system provided objective feedback as a referee to evaluate the performance of athletes, effectively improving the competitive level of athletes and reducing the controversy caused by subjective scoring. However, in real-life scenarios, these models still face some limitations that limit their widespread application.

[0003] The impact of environmental factors on action recognition cannot be ignored. In actual scenes, factors such as lighting changes and background complexity will significantly affect the video quality, which in turn affects the recognition and evaluation effects of the model. For example, direct sunlight or shadow areas in outdoor sports may lead to inaccurate human posture detection, affecting the evaluation results. Most of the existing AQA models do not fully consider these environmental factors, resulting in unstable performance in complex and changing environments. In addition, some actions may take a long time to complete, and these long actions often contain rich information. For example, each action in traditional fitness activities such as Ba Duan Jin takes a long time to complete, and the continuity and rhythm of the action are crucial to the overall effect. The existing system mainly focuses on the changes in human posture in a short period of time, and has limited ability to capture action features over a long span, which makes it difficult to fully capture the details and dynamic changes of the action during the evaluation, affecting the accuracy and comprehensiveness of the evaluation. Therefore, we propose a multi-person action quality evaluation based on contrast regression. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the shortcomings of the prior art, the present invention provides a multi-person action quality assessment based on contrastive regression, which introduces an environmental feature vector through an environmental adaptability learning module to improve the robustness and accuracy of the model in complex environments, and captures long-term action features through a time-aware regression module combined with a long-term memory mechanism to improve the accuracy and comprehensiveness of the assessment, and solves the problem that environmental factors, such as illumination changes and background complexity, significantly affect the recognition and evaluation effects of existing AQA models, especially the unstable performance in complex environments, and the existing models have limited ability to capture long-term action features, making it difficult to fully capture the details and dynamic changes of the action, affecting the accuracy and comprehensiveness of the assessment.

[0006] (II) Technical solution

[0007] In order to achieve the above-mentioned purpose of introducing environmental feature vectors through the environmental adaptability learning module to improve the robustness and accuracy of the model in complex environments, and capturing long-term action features through the time-aware regression module combined with the long-term memory mechanism to improve the accuracy and comprehensiveness of the evaluation, the present invention provides the following technical solutions: A multi-person action quality evaluation based on contrast regression, comprising the following steps:

[0008] S1. Obtain a multi-person Ba Duan Jin video dataset, and annotate the dataset after preprocessing;

[0009] S2, divide the entire video sequence V into n segments, each video segment contains the same number of video frames, representing continuous action snapshots, and perform posture detection on different segments to obtain single-person video segments and single-person posture segments, and finally perform feature extraction on different segments to obtain clip-level video features and posture F n ,k n ;

[0010] S3, introduce the environment adaptability learning module, extract the environment features of the video frame, fuse the environment features with the human posture features, and obtain the enhanced feature representation F;

[0011] S4, use the adaptive fusion module to select important channels for each feature, obtain the channel attention weight, assign weights to each feature and weight them to obtain the fusion feature G n ;

[0012] S5: Use the time-aware module to make fine-grained score predictions for each segment, and then perform weighted average of the segment scores to obtain the final evaluation score.

[0013] S6. Introduce a long-term memory mechanism, use LSTM and GRU units to store and update the historical state information of each individual, and combine multi-scale feature fusion to improve the ability to capture long-term action features;

[0014] S7. During model training, the same-scene comparison score regression module is used to highlight the subtle differences in multi-person features, predict the relative scores between two people, and use the loss function to optimize the overall model.

[0015] Preferably, the specific process of S1 includes the following steps:

[0016] S 1.1 : Obtain a multi-person Ba Duan Jin action quality assessment dataset, using AVA AX-C22P to record videos with a resolution of 1080p and a FPS of 25 frames;

[0017] S 1.2: Preprocess the motion video, filter out valid videos, remove incomplete videos, cut redundant videos irrelevant to the motion, and only keep the clips from the start to the end of the action;

[0018] S 1.3 : Construct a labeling file, and label each video in the order of person ID, action score, and the start and end time of each sub-action.

[0019] Preferably, the specific process of S2 includes the following steps:

[0020] S 2.1 :For a video with a total number of frames N, V ​​= {X1, X2, ..., X N}, the entire video sequence Divide into T segments, each video clip contains the same number of frames to obtain |V={V1,V2,...,V T};

[0021] S 2.2 : Use Alphapose multi-person posture detection algorithm to detect multiple people in the video, and obtain the local video clips of a single person and the single-person posture video clips in each video;

[0022] S 2.3 :Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video and posture segment to obtain clip-level video and posture features

[0023] Preferably, the specific process of S3 includes the following steps:

[0024] S 3.1 :Use the pre-trained CNN model to process each frame of video and extract the environmental feature vector E;

[0025] S 3.2 : The environment feature vector E is fused with the human posture feature vector P, which can be achieved by weighted summation or splicing to obtain the enhanced feature representation F:

[0026] F=αE+(1-α)P

[0027] Among them, α is a learnable weight parameter used to balance the importance of environmental features and posture features;

[0028] S 3.3 : During the training process, the environment feature vector E is used as an additional input to adjust the loss function It also takes into account the evaluation error of the action quality and environmental characteristics

[0029]

[0030] Among them, λ is a hyperparameter used to balance the weights of the two loss terms.

[0031] Preferably, the specific process of S4 includes the following steps:

[0032] S 4.1 :First, the information from the two branches is fused by element-wise addition, then the fusion information is embedded using global average pooling along the time dimension, and then the merged features are obtained through a simple fully connected layer

[0033] S 4.2 : Pass the merged features through a linear layer and use Softmax to get a soft attention map while matching the dimensions of the input feature size:

[0034] A=Softmax(MLP A (z)),B=Softmax(MLP B (z))

[0035] Finally, the attention maps are assigned to the corresponding features, and the elements are added to obtain the fused features.

[0036] Preferably, the specific process of S5 includes the following steps:

[0037] S 5.1 :Achieve fine-grained action evaluation and estimate the score of each video clip along the time dimension to obtain the score

[0038] S 5.2 : Aggregate the scores of all segments to get the final action prediction score

[0039] S 5.3 : During training, the MSE loss function is used to guide network training

[0040] Preferably, the specific process of S6 includes the following steps:

[0041] S 6.1 :Integrate one of the units in LSTM and GRU in the time-aware regression module to store and update the historical state information h of each individual t ;

[0042] h t =LSTM / GRU(xt ,h t-1 )

[0043] Among them, x t is the input feature of the current time step, h t-1 is the hidden state of the previous time step;

[0044] S 6.2 : Use the dynamic time warping (DTW) algorithm to align the action sequences of different individuals to ensure that the long-term memory unit can effectively capture the action characteristics across time;

[0045] S 6.3 : Combine the feature representations F under different time windows to form a multi-scale time-space feature representation F multi-scale :

[0046] F multi-scale =Concat(F1,F2,…,F n )

[0047] Among them, F i is the feature representation of the i-th time window;

[0048] S 6.4 : During the training process, the MSE loss function is used and time-based loss terms The combination of ensures that the model learns more stable long-term feature representations:

[0049]

[0050] Among them, γ is a hyperparameter used to balance the weights of the two loss terms.

[0051] Preferably, the specific process of S7 includes the following steps:

[0052] S 7.1 : In the model training stage, the model parameters are optimized by contrast regression module, and different individuals in the same picture are paired to obtain A team;

[0053] S 7.2 :Use motion difference decoder to extract the action difference in each team feature, and represent the team features as example features and target features G respectively example ,G target , map the example features to Q = W Q G example , map the target feature to IK = W K G target ,V=W V G target ,pass The difference feature representation F is obtained, where the negative sign is to capture the dissimilarity between features;

[0054] S 7.3 : After obtaining the difference features, the regression strategy with the same structure in step S5 is used to predict the relative score And also use the MSE loss function to calculate the loss to guide model training The total loss function of the network is

[0055] (III) Beneficial effects

[0056] Compared with the prior art, the present invention provides a multi-person action quality assessment based on contrast regression, which has the following beneficial effects:

[0057] 1. This multi-person action quality assessment based on contrast regression, the environmental adaptability learning module extracts the environmental feature vector of each frame of the video through a pre-trained convolutional neural network, and fuses it with the human posture feature vector to obtain an enhanced feature representation. During the training process, the environmental feature vector is used as an additional input, and the loss function is adjusted to simultaneously consider the assessment error of the action quality and the influence of environmental characteristics. By introducing the environmental adaptability learning module, the model can more stably identify and evaluate actions in complex and changeable environments, improve robustness and accuracy. For example, in outdoor sports, direct sunlight or shadow areas may cause inaccurate human posture detection, but the introduction of environmental features can make up for these shortcomings and improve the performance of the model.

[0058] 2. This multi-person action quality assessment based on contrastive regression, the time-aware regression module integrates long short-term memory or gated recurrent units to store and update the historical state information of each individual, combines the feature representations under different time windows, and forms a multi-scale time-space feature representation. The dynamic time warping algorithm is used to align the action sequences of different individuals to ensure that the long-term memory unit can effectively capture the action features across time. During the training process, a combination of the mean square error loss function and the time-based loss term is used to ensure that the model learns a more stable long-term feature representation. By introducing the time-aware regression module and the long-term memory mechanism, the model can better capture the action features of a long time span and improve the accuracy and comprehensiveness of the evaluation. For example, in traditional fitness activities such as Ba Duan Jin, each action takes a long time to complete, and the coherence and rhythm of the action are crucial to the overall effect. The introduction of these modules enables the model to capture the details and dynamic changes of the action more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 is a flow chart of the present invention;

[0060] Figure 2It is a structural diagram of the motion difference decoder of the present invention. DETAILED DESCRIPTION

[0061] The following will be combined with the embodiments of the present invention and the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] See also Figure 1-2 , a multi-person action quality assessment based on contrast regression, characterized by comprising the following steps:

[0063] S1. Obtain a multi-person Ba Duan Jin video dataset, and annotate the dataset after preprocessing:

[0064] S 1.1 : Obtain a multi-person Ba Duan Jin action quality assessment dataset, using AVA AX-C22P to record videos with a resolution of 1080p and a FPS of 25 frames;

[0065] S 1.2 : Preprocess the motion video, filter out valid videos, remove incomplete videos, cut redundant videos irrelevant to the motion, and only keep the clips from the start to the end of the action;

[0066] S 1.3 : Construct a labeling file, and label each video in the order of person ID, action score, and the start and end time of each sub-action.

[0067] S2, divide the entire video sequence V into n segments, each video segment contains the same number of video frames, representing continuous action snapshots, and perform posture detection on different segments to obtain single-person video segments and single-person posture segments, and finally perform feature extraction on different segments to obtain clip-level video features and posture F n ,k n :

[0068] S 2.1 :For a video with a total number of frames N, V ​​= {X1, X2, ..., X N}, the entire video sequence Divide into T segments, each video clip contains the same number of frames to obtain |V={V1,V2,…,V T};

[0069] S 2.2: Use Alphapose multi-person posture detection algorithm to detect multiple people in the video, and obtain the local video clips of a single person and the single-person posture video clips in each video;

[0070] S 2.3 :Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video and posture segment to obtain clip-level video and posture features

[0071] S3, introduce the environment adaptability learning module, extract the environment features of the video frame, fuse the environment features with the human posture features, and obtain the enhanced feature representation F:

[0072] S 3.1 :Use the pre-trained CNN model to process each frame of video and extract the environmental feature vector E;

[0073] S 3.2 : The environment feature vector E is fused with the human posture feature vector P, which can be achieved by weighted summation or splicing to obtain the enhanced feature representation F:

[0074] F=αE+(1-α)P

[0075] Among them, α is a learnable weight parameter used to balance the importance of environmental features and posture features;

[0076] S 3.3 : During the training process, the environment feature vector E is used as an additional input to adjust the loss function It also takes into account the evaluation error of the action quality and environmental characteristics

[0077]

[0078] Among them, λ is a hyperparameter used to balance the weights of the two loss terms.

[0079] S4, use the adaptive fusion module to select important channels for each feature, obtain the channel attention weight, assign weights to each feature and weight them to obtain the fusion feature G n :

[0080] S 4.1 :First, the information from the two branches is fused by element-wise addition, then the fusion information is embedded using global average pooling along the time dimension, and then the merged features are obtained through a simple fully connected layer

[0081] S 4.2: Pass the merged features through a linear layer and use Softmax to get a soft attention map while matching the dimensions of the input feature size:

[0082] A=Softmax(MLP A (z)),B=Softmax(MLP B (z))

[0083] Finally, the attention maps are assigned to the corresponding features, and the elements are added to obtain the fused features.

[0084] S5: Use the time-aware module to make fine-grained score predictions for each segment, and then perform weighted average of the segment scores to obtain the final evaluation score.

[0085] S 5.1 :Achieve fine-grained action evaluation and estimate the score of each video clip along the time dimension to obtain the score

[0086] S 5.2 : Aggregate the scores of all segments to get the final action prediction score

[0087] S 5.3 : During training, the MSE loss function is used to guide network training

[0088] S6. Introduce a long-term memory mechanism, use LSTM and GRU units to store and update the historical state information of each individual, and combine multi-scale feature fusion to improve the ability to capture long-term action features:

[0089] S 6.1 :Integrate one of the units in LSTM and GRU in the time-aware regression module to store and update the historical state information h of each individual t ;

[0090] h t =LSTM / GRU(x t ,h t-1 )

[0091] Among them, x t is the input feature of the current time step, h t-1 is the hidden state of the previous time step;

[0092] S 6.2 : Use the dynamic time warping (DTW) algorithm to align the action sequences of different individuals to ensure that the long-term memory unit can effectively capture the action characteristics across time;

[0093] S 6.3 : Combine the feature representations F under different time windows to form a multi-scale time-space feature representation F multi-scale :

[0094] F multi-scale =Concat(F1, F2, ..., F n )

[0095] Among them, F i is the feature representation of the i-th time window;

[0096] S 6.4 : During the training process, the MSE loss function is used and time-based loss terms The combination of ensures that the model learns more stable long-term feature representations:

[0097]

[0098] Among them, γ is a hyperparameter used to balance the weights of the two loss terms.

[0099] S7. During model training, the same-scene comparison score regression module is used to highlight the subtle differences in multi-person features, predict the relative scores between two people, and use the loss function to optimize the overall model:

[0100] S 7.1 : In the model training stage, the model parameters are optimized by contrast regression module, and different individuals in the same picture are paired to obtain A team;

[0101] S 7.2 :Use motion difference decoder to extract the action difference in each team feature, and represent the team features as example features and target features G respectively example , G target , map the example features to Q = W Q G example , map the target feature to IK = W K G target , V=W V G target ,pass The difference feature representation F is obtained, where the negative sign is to capture the dissimilarity between features;

[0102] S 7.3 : After obtaining the difference features, the regression strategy with the same structure in step S5 is used to predict the relative score And also use the MSE loss function to calculate the loss to guide model training The total loss function of the network is

[0103] The multi-person action quality assessment based on contrast regression described in the above steps S1 to S7 is applied to a specific example below, and its technical effect is further demonstrated through an embodiment.

[0104] Data source display:

[0105] Dataset: It consists of 192 video samples, including 82 frontal videos and 84 30° side videos. The average duration of each video is 4 minutes and 43 seconds (283 seconds), which is one of the longest action quality assessment datasets currently.

[0106] Video content: Each video sample contains multiple children exercising. The most common video contains 5 children, followed by 3 or 4 children. In order to enhance the challenge of the model, 4 groups of 7 children were recorded.

[0107] Annotation tool: Use professional software to annotate the video, including the person ID, action score, and the start and end time of each sub-action;

[0108] Computing resources: GPU server equipped with NVIDIA Tesla V100 GPU, 16GB video memory, 64GB memory, IntelXeon E5-2690 v4 CPU.

[0109] Experimental steps:

[0110] Step 1: Data collection and preprocessing:

[0111] Collect 192 video samples, including frontal and side 30° videos; pre-process the videos, including screening valid videos, removing incomplete videos, cutting redundant videos irrelevant to the motion, and only retaining the segments from the start to the end of the action; construct a labeling file, and label each video in the order of person ID, action score, and the start and end time of each sub-action;

[0112] Step 2: Data division:

[0113] The dataset is divided into training set (70%), validation set (15%) and test set (15%).

[0114] Step 3: Feature extraction:

[0115] Each video sequence is divided into 10 segments, each of which contains the same number of video frames; each video segment is detected using the Alphapose multi-person posture detection algorithm to obtain single-person video segments and single-person posture video segments; the Video Swine Transformer (VST) feature extraction module is used to extract 1024-dimensional video features from each video and posture segment to obtain clip-level video and posture features;

[0116] Step 4: Environmental feature extraction and fusion:

[0117] Use the pre-trained CNN model to process each frame of video and extract the environmental feature vector; fuse the environmental feature vector with the human posture feature vector, which can be achieved by weighted summation or splicing to obtain an enhanced feature representation; during the training process, use the environmental feature vector as an additional input and adjust the loss function to take into account both the evaluation error of the action quality and the influence of environmental characteristics;

[0118] Step 5: Model training:

[0119] The adaptive fusion module is used to select important channels for each feature. After obtaining the channel attention weight, each feature is assigned a weight and weighted to obtain the fused feature. The time perception module is used to predict the fine-grained score of each segment, and then the segment scores are weighted averaged to obtain the final evaluation score. The long-term memory mechanism is introduced, and the historical state information of each individual is stored and updated using LSTM and GRU units. The multi-scale feature fusion is combined to improve the ability to capture long-term action features. During model training, the contrast regression module is used to highlight the subtle differences in multi-person features, predict the relative scores between two people, and use the loss function to optimize the overall model.

[0120] Step 6: Model Evaluation

[0121] The performance of the model was evaluated on the test set, with the main indicators including the Spearman rank correlation coefficient (ρ) and relative L2 distance (〖Rl〗_2); the performance of the model in capturing features of different view angles, different numbers of people and long-term motion was analyzed; the performance of the model under different conditions was compared, especially the performance of front and side view 30° videos and different numbers of people.

[0122] Experimental results and conclusions:

[0123] Performance indicators - Table 1

[0124] Model ρ 〖Rl〗_2(×100) MLP 0.8263 1.35 GDLT 0.8380 2.25 USDL 0.8334 3.37 DAE 0.8353 1.52 PCLN 0.8261 1.51 Baseline of the present invention 0.8528 1.25 The present invention 0.8596 1.24

[0125] Performance at different viewing angles - Table 2

[0126] Perspective ρ 〖Rl〗_2(×100) face 0.8610 1.23 Side view 30° 0.8582 1.25

[0127] Performance of different numbers of people - Table 3

[0128] Number of people ρ 〖Rl〗_2(×100) 3 people 0.8550 1.26 4 people 0.8570 1.24 5 people 0.8600 1.23 7 people 0.8560 1.25

[0129] Long-term motion feature capture - Table 4

[0130] Action Type ρ 〖Rl〗_2(×100) Ba Duan Jin 0.8620 1.22

[0131] Experimental conclusion:

[0132] The test results are shown in Tables 1-4. The results show that the multi-person motion quality assessment based on contrast regression proposed in the present invention: the multi-person motion quality assessment system based on contrast regression performs well in capturing motion features from different perspectives, different numbers of people and for a long time, especially in complex environments and high-density crowds; by introducing the same-scene contrast regression module, the Spearman rank correlation coefficient (ρ) and relative L2 distance (〖Rl〗_2) of the model are significantly improved; it has high practical value in practical applications, especially in the fields of sports training, fitness guidance, etc.

[0133] In summary, the multi-person action quality assessment based on contrast regression, the environmental adaptability learning module extracts the environmental feature vector of each frame of the video through a pre-trained convolutional neural network, and fuses it with the human posture feature vector to obtain an enhanced feature representation. During the training process, the environmental feature vector is used as an additional input, and the loss function is adjusted to simultaneously consider the assessment error of the action quality and the influence of environmental characteristics. By introducing the environmental adaptability learning module, the model can more stably identify and evaluate actions in complex and changeable environments, improve robustness and accuracy. For example, in outdoor sports, direct sunlight or shadow areas may cause inaccurate human posture detection, but the introduction of environmental features can make up for these shortcomings and improve the performance of the model.

[0134] In this multi-person action quality assessment based on contrastive regression, the time-aware regression module integrates long-term and short-term memory or gated recurrent units to store and update the historical state information of each individual, combines the feature representations under different time windows, and forms a multi-scale time-space feature representation. The dynamic time warping algorithm is used to align the action sequences of different individuals to ensure that the long-term memory unit can effectively capture the action features across time. During the training process, a combination of the mean square error loss function and the time-based loss term is used to ensure that the model learns a more stable long-term feature representation. By introducing the time-aware regression module and the long-term memory mechanism, the model can better capture the action features of a long time span and improve the accuracy and comprehensiveness of the evaluation. For example, in traditional fitness activities such as Ba Duan Jin, each action takes a long time to complete, and the coherence and rhythm of the action are crucial to the overall effect. The introduction of these modules enables the model to more comprehensively capture the details and dynamic changes of the action, solving the problem that environmental factors such as illumination changes and background complexity significantly affect the recognition and evaluation effects of existing AQA models, especially in complex environments. The performance is unstable, and the existing model has limited ability to capture long-term action features, making it difficult to fully capture the details and dynamic changes of the action, affecting the accuracy and comprehensiveness of the evaluation.

[0135] The relevant modules involved in this system are all hardware system modules or functional modules that combine computer software programs or protocols with hardware in the prior art. The computer software programs or protocols involved in the functional modules are themselves technologies that are well known to those skilled in the art and are not improvements of this system. The improvements of this system are the interaction or connection relationships between the modules, that is, improvements to the overall structure of the system to solve the corresponding technical problems to be solved by this system.

[0136] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-person action quality assessment based on contrastive regression, characterized in that: The following steps are involved: S1. Obtain a multi-person Ba Duan Jin video dataset, and annotate the dataset after preprocessing; S2, divide the entire video sequence V into n segments, each video segment contains the same number of video frames, representing continuous action snapshots, and perform posture detection on different segments to obtain single-person video segments and single-person posture segments, and finally perform feature extraction on different segments to obtain clip-level video features and posture F n ,k n ; S3, introduce the environment adaptability learning module, extract the environment features of the video frame, fuse the environment features with the human posture features, and obtain the enhanced feature representation F; S4, use the adaptive fusion module to select important channels for each feature, obtain the channel attention weight, assign weights to each feature and weight them to obtain the fusion feature G n ; S5: Use the time-aware module to make fine-grained score predictions for each segment, and then perform weighted average of the segment scores to obtain the final evaluation score. S6. Introduce a long-term memory mechanism, use LSTM and GRU units to store and update the historical state information of each individual, and combine multi-scale feature fusion to improve the ability to capture long-term action features; S7. During model training, the same-scene comparison score regression module is used to highlight the subtle differences in multi-person features, predict the relative scores between two people, and use the loss function to optimize the overall model.

2. The multi-person action quality assessment based on contrast regression according to claim 1, characterized in that: The specific process of S1 includes the following steps: S 1.1 : Obtain a multi-person Ba Duan Jin action quality assessment dataset, using AVA AX-C22P to record videos with a resolution of 1080p and a FPS of 25 frames; S 1.2 : Preprocess the motion video, filter out valid videos, remove incomplete videos, cut redundant videos irrelevant to the motion, and only keep the clips from the start to the end of the action; S 1.3 : Construct a labeling file, and label each video in the order of person ID, action score, and the start and end time of each sub-action.

3. The multi-person action quality assessment based on contrast regression according to claim 1, characterized in that: The specific process of S2 includes the following steps: S 2.1 :For a video with a total number of frames N, V ​​= {X1, X2, ..., X N }, the entire video sequence Divide into T segments, each video clip contains the same number of frames to obtain |V={V1,V2,…,V T }; S 2.2 : Use Alphapose multi-person posture detection algorithm to detect multiple people in the video, and obtain the local video clips of a single person and the single-person posture video clips in each video; S 2.3 :Use the feature extraction module Video Swin Transformer (VST) to extract 1024-dimensional video features for each video and posture segment to obtain clip-level video and posture features 4. The method for multi-person action quality assessment based on contrastive regression according to claim 1, characterized in that: The specific process of S3 includes the following steps: S 3.1 :Use the pre-trained CNN model to process each frame of video and extract the environmental feature vector E; S 3.2 : The environment feature vector E is fused with the human posture feature vector P, which can be achieved by weighted summation or splicing to obtain the enhanced feature representation F: F=αE+(1-α)P Among them, α is a learnable weight parameter used to balance the importance of environmental features and posture features; S 3.3 : During the training process, the environment feature vector E is used as an additional input to adjust the loss function It also takes into account the evaluation error of the action quality and environmental characteristics Among them, λ is a hyperparameter used to balance the weights of the two loss terms.

5. The multi-person action quality assessment based on contrast regression according to claim 1, characterized in that: The specific process of S4 includes the following steps: S 4.1 :First, the information from the two branches is fused by element-wise addition, then the fusion information is embedded using global average pooling along the time dimension, and then the merged features are obtained through a simple fully connected layer S 4.2 : Pass the merged features through a linear layer and use Softmax to get a soft attention map while matching the dimensions of the input feature size: A=Softmax(MLP A (z)),B=Softmax(MLP B (z)) Finally, the attention maps are assigned to the corresponding features, and the elements are added to obtain the fused features.

6. The method for multi-person action quality assessment based on contrastive regression according to claim 1, characterized in that: The specific process of S5 includes the following steps: S 5.1 :Achieve fine-grained action evaluation and estimate the score of each video clip along the time dimension to obtain the score S 5.2 : Aggregate the scores of all segments to get the final action prediction score S 5.3 : During training, the MSE loss function is used to guide network training 7. The method for multi-person action quality assessment based on contrastive regression according to claim 1, characterized in that: The specific process of S6 includes the following steps: S 6.1 :Integrate one of the units in LSTM and GRU in the time-aware regression module to store and update the historical state information h of each individual t ; h t =LSTM / GRU(x t ,h t-1 ) Among them, x t is the input feature of the current time step, h t-1 is the hidden state of the previous time step; S 6.2 : Use the dynamic time warping (DTW) algorithm to align the action sequences of different individuals to ensure that the long-term memory unit can effectively capture the action characteristics across time; S 6.3 : Combine the feature representations F under different time windows to form a multi-scale time-space feature representation F multi-scale : F multi-scale =Concat(F1,F2,…,F n ) Among them, F i is the feature representation of the i-th time window; S 6.4 : During the training process, the MSE loss function is used and time-based loss terms The combination of ensures that the model learns more stable long-term feature representations: Among them, γ is a hyperparameter used to balance the weights of the two loss terms.

8. The method for multi-person action quality assessment based on contrastive regression according to claim 1, characterized in that: The specific process of S7 includes the following steps: S 7.1 : In the model training stage, the model parameters are optimized by contrast regression module, and different individuals in the same picture are paired to obtain A team; S 7.2 :Use motion difference decoder to extract the action difference in each team feature, and represent the team features as example features and target features G respectively axample ,G target , map the example features to Q = W Q G example , map the target feature to IK = W K G target ,V=W V G target ,pass The difference feature representation F is obtained, where the negative sign is to capture the dissimilarity between features; S 7.3 : After obtaining the difference features, the regression strategy with the same structure in step S5 is used to predict the relative score And also use the MSE loss function to calculate the loss to guide model training The total loss function of the network is

Images