Pull-up action quality assessment method, computer device and storage medium

Abstract

The present invention belongs to the technical field of human motion quality assessment, and specifically discloses a pull-up action quality assessment method, a computer device, and a storage medium. The method of the present invention divides the pull-up action video into three key stages: body rise, head and chin passing the bar, and body descent by designing the segmentation and action quality scoring standards of the pull-up action video, comprehensively considers the completion degree of each stage, and weightedly calculates the final score of the action completion quality. In addition, the present invention also constructs a pull-up action quality assessment model based on Transformer. The model adopts a dual-stream network structure, which is used to capture the dynamic and static features of the pull-up action and fuse them respectively, and the multi-stage attention network in the decoder component assigns a weight coefficient to each action stage. The present invention well realizes the automated and accurate evaluation of the pull-up action, and improves the evaluation efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of human motion quality assessment, and in particular relates to a pull-up motion quality assessment method, a computer device and a storage medium. Background Art

[0002] At present, there are two main ways to detect and evaluate human motion using information technology. One is to collect and analyze human motion data based on positioning sensors. Positioning sensors are placed at key parts of the human body, and the inertial navigation parameters of the sensors are used to track the real-time positions of the corresponding parts to achieve detection and recognition of human motion. This method has high positioning accuracy and good sensitivity. However, the equipment cost is high and the subjects will be constrained and interfered by the sensor device when performing actions, which affects the continuity of the action. The other is to use single-camera or multi-camera camera equipment to collect action videos, and to evaluate the actions by annotating and analyzing the videos. This method has become the method adopted by most video-assisted refereeing systems for sports competitions because of its simple and easy signal acquisition and good acceptability for subjects. However, the action quality evaluation based on video analysis and understanding will be affected by the camera environment and algorithm adaptability, and has high requirements for the robustness and generalization ability of the algorithm or prediction model.

[0003] The motion quality assessment methods mainly use score regression, grade classification and quality grade ranking. In recent years, deep learning has gradually become the mainstream technology for video motion quality assessment. For example, Zhang Yu et al. used label distribution learning and spatiotemporal attention awareness for video motion quality assessment. By focusing on the different spatiotemporal position motion information in the video sequence, fine-grained labels are generated, which effectively solves the problem of label ambiguity. Li et al. proposed a skeleton-based spatiotemporal posture extraction module and used a recurrent neural network to interactively extract the dynamic temporal structure of the skeleton sequence to achieve accurate assessment of the quality of skating movements.

[0004] Due to the large number of sports events, the influence of shooting environment interference and action complexity, the action quality assessment based on video analysis and understanding still needs to be further explored. At present, the research on automatic assessment methods for physical test items is basically blank.

[0005] Pull-ups are an important physical test subject and are widely used in the fields of physical education and physical fitness assessment. Currently, the scoring of pull-ups mostly relies on manual methods. The scorers need to accurately record the details and number of movements of each person being evaluated, which undoubtedly increases the difficulty and complexity of scoring. Manual scoring is easily affected by subjective factors, such as the distraction and fatigue of the assessors, which may affect the fairness and accuracy of the scoring. At present, the application of action quality assessment models is mainly concentrated in large-scale sports events such as diving, skating, gymnastics, and martial arts. These public data sets are large, and their models are not suitable for small data sets such as pull-ups. In addition, there is no public pull-up data set yet. Summary of the invention

[0006] The purpose of the present invention is to propose a pull-up action quality assessment method, make full use of the advantages of Transformer in processing time series data, and construct a pull-up action quality assessment model based on Transformer to achieve automated and accurate scoring of pull-up actions, which is conducive to improving scoring efficiency and saving manpower and material resources.

[0007] In order to achieve the above object, the present invention adopts the following technical scheme:

[0008] A method for evaluating the quality of a pull-up movement comprises the following steps:

[0009] Step 1. Design a phased action normative evaluation standard and scoring method based on the characteristics of pull-ups; divide the video action sequence of pull-ups into three key stages: body rising, head and chin passing the bar, and body descending;

[0010] Step 2. Build a Transformer-based pull-up action quality assessment model, which includes a feature extraction network with a dual-branch structure, a feature fusion network, and a score regression module;

[0011] The feature extraction network with a dual-branch structure includes a dynamic flow feature extraction network branch and a static flow feature extraction network branch, and is used to extract dynamic flow output features and static flow output features respectively;

[0012] The dynamic flow output features and the static flow output features are fused in the feature fusion network, and the fused results are sent to the score regression module to obtain the predicted score, and the pull-up action quality evaluation result is given according to the predicted score.

[0013] In addition, based on the pull-up action quality assessment method, the present invention also proposes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory. When the processor executes the executable code, it is used to implement the pull-up action quality assessment method.

[0014] In addition, based on the pull-up action quality assessment method, the present invention also proposes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the pull-up action quality assessment method.

[0015] The present invention has the following advantages:

[0016] As described above, the present invention relates to a pull-up action quality assessment method. By designing the segmentation and action quality scoring criteria of the pull-up action video, the pull-up action video is divided into three key stages: body rise, head chin over the bar and body descent. The degree of completion of each stage is comprehensively considered and the final score of the action completion quality is weighted. In addition, the present invention also makes full use of the advantages of Transformer in processing time series data, and constructs a pull-up action quality assessment model based on Transformer, wherein the model adopts a dual-stream network structure to capture the dynamic and static features of the pull-up action respectively and fuse them, and the multi-stage attention network in the decoder component assigns weight coefficients to each action stage. The pull-up action quality assessment model based on Transformer can capture the key features and time dependence of the action through its self-attention mechanism and sequence modeling ability, which is conducive to improving the prediction accuracy of action quality assessment. The method of the present invention well realizes the automatic and accurate scoring of the pull-up action, improves the efficiency of action quality assessment, and saves manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Flow chart of a method for evaluating the quality of pull-up movements in an embodiment of the present invention;

[0018] Figure 2 illustrative diagram of standard / non-standard movements at various stages of a pull-up in an embodiment of the present invention;

[0019] Figure 3 The figure is a general architecture diagram of a Transformer-based pull-up action quality assessment model in an embodiment of the present invention;

[0020] Figure 4 is a structural diagram of a dynamic feature extractor in an embodiment of the present invention;

[0021] Figure 5 is a structural diagram of a static flow feature extractor in an embodiment of the present invention;

[0022] Figure 6 A structural diagram of a context-aware encoder according to an embodiment of the present invention;

[0023] Figure 7 2 is a structural diagram of a multi-stage attention network in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0025] Example 1

[0026] This embodiment describes a Transformer-based pull-up action quality assessment method, which fully extracts features from a pull-up action video to achieve accurate and automated assessment of the pull-up action quality.

[0027] The method of the present invention first designs a pull-up physical test action quality assessment method, gives the real score supervision of each action, then uses the temporal characteristics of Transformer to build a special action quality assessment model for pull-ups, and finally feeds the action video into the action quality assessment model. This method can effectively solve the problem of time-consuming and labor-intensive manual operation, avoid the influence of the subjectivity of the scorer, and thus improve the accuracy and efficiency of the pull-up action assessment.

[0028] like Figure 1 As shown, the pull-up action quality assessment method in this embodiment specifically includes the following steps:

[0029] Step 1. According to the characteristics of pull-ups, design phased action normative evaluation criteria and scoring methods; divide the pull-ups video action sequence into three key stages: torso rising, head and chin passing the bar, and torso descending.

[0030] The present invention combines the "National Student Physical Fitness Standard (2014 Revised)" promulgated by the Ministry of Education in 2014 and the movement characteristics of the pull-up, and considers the impact of obvious non-standard movements on the completion quality, and designs a scoring method with staged weighted scoring and reasonable deduction of points. The standards are shown in Table 1.

[0031] Table 1 Pull-up physical test scoring method based on video analysis and understanding

[0032]

[0033] The entire pull-up action video is divided into three key stages, namely, the torso rising stage, the chin passing the bar stage, and the torso descending stage, and weight coefficients are assigned to the corresponding key stages, for example, 0.4, 0.3, and 0.3 respectively.

[0034] According to the proportion of the distribution weight coefficient, if the total score of the movement quality is 5 points, the full score of the torso rising phase is 2 points, the full score of the chin-over-the-bar phase is 1.5 points, and the full score of the torso descending phase is 1.5 points.

[0035] During the stage of rising the body, if the arms are not straightened, the body swings or the legs are bent at the start, 1.0, 0.2 and 0.2 points will be deducted respectively; during the stage of chin-over-bar, if the legs are bent and the chin is not over the bar, 0.2 points and 1.0 points will be deducted respectively; during the stage of descending the body, if the legs are close together and the arms are not straightened when lowering, 0.2 points and 1.0 points will be deducted respectively.

[0036] Example pictures of some standard and non-standard actions in each stage above and enlarged pictures of local actions, such as Figure 2 shown.

[0037] Step 2. Build the Transformer-based pull-up action quality assessment model DS-MSAT, such as Figure 3 shown.

[0038] The pull-up action quality assessment model DS-MSAT is based on the Transformer backbone, which includes a feature extraction network with a dual-branch structure, a feature fusion network, and a score regression module.

[0039] Among them, the feature extraction network with a dual-branch structure includes a dynamic flow feature extraction network branch and a static flow feature extraction network branch, and is used to extract dynamic flow output features and static flow output features respectively.

[0040] Specifically, the dynamic flow feature extraction network branch includes a dynamic flow feature extractor, a context-aware encoder, and a multi-stage attention decoder; the processing flow of the input video frame in the dynamic flow feature extraction network branch is as follows:

[0041] The dynamic flow feature extractor first extracts dynamic information, that is, captures and analyzes the motion changes of adjacent frames in the video frame sequence, identifies the posture and position changes between each frame during the pull-up process, learns the timing information of the three stages of torso rising, head and chin passing the bar, and torso descending, and obtains dynamic flow features. , the dynamic flow features are input into the context-aware encoder, the correlation features between frames are calculated, and the dynamic flow enhancement features are obtained , dynamic flow enhancement features and feature prototypes of the three action stages of dynamic flow P D (m) is simultaneously input into the multi-stage attention decoder, the complete video of the pull-up action is divided into key stages, and then the corresponding weight coefficient is assigned to each key stage and weighted processing is performed to obtain the dynamic flow output feature F D .

[0042] The static stream feature extraction network branch includes a static stream feature extractor, a context-aware encoder, and a multi-stage attention decoder. The processing flow of the input video frame in the static stream feature extraction network branch is as follows:

[0043] The static flow feature extractor first extracts posture information, i.e., the position, posture, and spatial relationship of each part of the human body in each frame, to obtain the static flow feature , the static flow features are input into the context-aware encoder, and the correlation features between frames are calculated to obtain the static flow enhancement features , static flow enhancement features and feature prototypes of the three action stages of static flow P S (m) is simultaneously input into the multi-stage attention decoder, the complete video of the pull-up action is divided into key stages, and then the corresponding weight coefficient is assigned to each key stage and weighted processing is performed to obtain the static flow output feature F S .

[0044] Among them, the characteristic prototype P D (m) and P S (m) refers to the representative standard features learned during the model training process by observing a large number of pull-up action samples.

[0045] Since the entire pull-up action video is divided into three key stages in this embodiment, the value of m is 1, 2, and 3.

[0046] The dynamic flow output features and the static flow output features are fused in the feature fusion network, and the fused results are sent to the score regression module to obtain the predicted score, and the pull-up action quality evaluation result is given according to the predicted score.

[0047] The dynamic flow feature extractor is used to extract the dynamic flow features of the input video. The dynamic feature extractor uses an inflated 3D convolutional neural network (I3D). I3D directly expands the 2D convolution kernel of size n×n to a 3D convolution kernel of size n×n×n by adding a time dimension to the trained 2DCNN. Figure 4 shown.

[0048] The video dimension input to the dynamic feature extractor is 16×256×256×3, and multiple convolution kernels and pooling operations of different sizes are cascaded as output in the I3D network. The I3D network includes a convolution structure, a pooling layer, and an Inception module.

[0049] The signal processing flow is as follows: in the feature extraction process, the input 16 256×256 frames first pass through a 7×7 3D convolution layer and a 3D maximum pooling layer, and the feature dimension obtained is 8×64×64×64; then, through a 1×1 3D convolution layer, a 3×3 3D convolution layer and a 3D maximum pooling layer, the dimension is adjusted to 8×32×32×192; then, after passing through two Inception modules and a 3D maximum pooling layer, the feature dimension is 4×16×16×482; further through five Inception modules and a 3D maximum pooling layer, the dimension becomes 2×8×8×832; finally, after passing through two Inception modules, a 2×8×8×1024 feature map is obtained, and through a global average pooling layer and a 3D average pooling layer, the dimension is reduced to 1×1×1024; throughout the process, the I3D network gradually extracts and compresses spatiotemporal features through multi-layer convolution and pooling operations, and uses the Inception module to further extract multi-scale features, and finally obtains a high-dimensional global feature representation.

[0050] The static stream feature extractor is used to extract the static features of the input video. It uses Resnet-50 (Deep Residual Network). The Resnet-50 network is a Resnet model with a depth of 50. Figure 5 shown.

[0051] The input image dimension of the Resnet model is 256×256×3, and the 7×7 three-dimensional convolution layer means that the convolution kernel is 7×7, the output channel is 64, the stride is 2, and there are four groups of residual blocks, each group contains 3, 4, 6, and 3 residual blocks, and each residual block contains three convolution layers. ResNet-50 includes convolution layer, batch normalization (BN), activation function (ReLU), maximum pooling layer, residual block (ResBlock) and fully connected layer (FC).

[0052] The signal processing flow is as follows: in the feature extraction process, the 16 input frames of size 256×256 first pass through a 7×7 3D convolution layer to obtain a feature dimension of 128×128×64; then, after batch normalization and ReLU activation function, and then through the maximum pooling layer, the feature dimension is adjusted to 64×64×64; then, through 3 residual blocks, the feature dimension becomes 64×64×256; after 4 residual blocks, the dimension is further reduced to 32×32×512; then, through 6 residual blocks, the dimension becomes 16×16×1024; finally, through 3 residual blocks, the feature dimension is adjusted to 8×8×2048; after the ReLU activation function, the feature dimension is compressed to 1×1×2048 using the global average pooling layer, and then through the fully connected layer to obtain the final feature output 1×1×1024. During the whole process, ResNet-50 effectively alleviated the gradient vanishing problem in deep networks by introducing residual connections, thereby improving the training efficiency and feature extraction capabilities of the model.

[0053] Although the I3D network and ResNet-50 model can extract high-quality spatial information in dynamic and static streams, it is difficult to capture the temporal and spatial correlation of actions. To this end, the present invention uses a context-aware encoder CAE to improve the ability to extract detailed features, such as Figure 6 Specifically, the processing of the context-aware encoder is as follows:

[0054] First, for the i-th input video, the self-attention mechanism is used to perform weighted aggregation on the initial features of its T-frame images to obtain context information; then, the feature vector of the context information is added to the initial feature vector, summed and normalized, and then passed to the feedforward neural network for fusion; finally, the enhanced features are obtained.

[0055] The formula for calculating the static flow features using the context-aware encoder is as follows:

[0056] { F = n o r m [ { f t i } t = 1 T + { f S i t } t = 1 T ] { f ′ S i t } t = 1 T = n o r m [ F + D N N θ x ( F ) ] (1)

[0057] In the formula, A feature vector representing context information, represents the initial feature vector of the static flow extracted by the Resnet-50 network, represents layer normalization, F represents the video features after addition and normalization, represents the enhanced features obtained after being processed by the context-aware encoder in the static stream; Indicates that the parameter is Feedforward neural network;

[0058] The formula for calculating the dynamic flow features using the context-aware encoder is as follows:

[0059] { F = n o r m [ { f t i } t = 1 T + { f D i t } t = 1 T ] { f ′ D i t } t = 1 T = n o r m [ F + D N N θ x ( F ) ] (2)

[0060] In the formula, A feature vector representing context information, represents the initial feature vector of the dynamic flow extracted by the I3D network, represents layer normalization, F represents the video features after addition and normalization, represents the enhanced features obtained after being processed by the context-aware encoder in the dynamic stream; Indicates that the parameter is Feedforward neural network.

[0061] As mentioned above, for the input video V i , the dynamic flow feature extraction network branch generates enhanced features with dynamic information by the dynamic feature extractor and the context-aware encoder The static flow feature extraction network branch generates enhanced features with static information by using the static feature extractor and the context-aware encoder. .

[0062] In order to automatically divide the three action stages of pull-ups and assign corresponding weight coefficients to each stage, a multi-stage attention decoder MSAD is used to decode the enhanced features of dynamic and static flows. The multi-stage attention decoder is an improvement on the traditional Transformer decoder, which proposes an attention unit to form a multi-stage attention network, such as Figure 7 shown.

[0063] The attention unit consists of two fully connected layers, a rectified linear unit (ReLU) and an activation function (softmax), which can obtain the weight coefficients of different action stages. D (m) and P S (m) represents the feature prototype (query feature) of the mth action stage in the dynamic flow and static flow branches respectively, m∈[1, M], and M is 3 in this embodiment.

[0064] The present invention allocates weights to different action stages based on M learnable feature vectors using a DETR-like approach.

[0065] First, masked multi-head attention is used to mine the relationship between the feature prototypes of the M action stages, and the updated feature prototype is recorded as and .

[0066] Then, multi-head attention is used to mine enhanced features and , The relationship between and Perform a linear projection to transform it into the corresponding query vector:

[0067] (3)

[0068] In the formula, represents the feature prototype of the mth action stage in the dynamic flow, Represents the feature prototype of the mth action stage in the static flow; is the projection parameter matrix of the dynamic flow query vector, is the projection parameter matrix of the static flow query vector. It is the enhanced features generated in dynamic flow or static flow, that is, or , represents the dynamic or static features of the tth frame of the input video. The key K and value V are given by Transformations are done via different linear mappings:

[0069] (4)

[0070] In the formula, represents the K vector of the tth frame, V vector representing the tth frame; is the projection parameter matrix of the key vector, is the projection parameter matrix of the value vector; the semantic relevance between the feature prototype of the mth action stage and the tth frame of the video is measured by the dot product similarity between the corresponding query vector q and the key k To measure:

[0071] (5)

[0072] In the formula, Represents the scale factor, divided by To prevent gradient explosion; That is or The query vector representing the mth action stage in a dynamic or static flow, symbol Represents the transpose of a vector.

[0073] The aggregated features of the m-th action stage are normalized by the coefficients The linear combination with the value V of the action phase frame is obtained:

[0074] f a g g ( m ) = ∑ t = m s m e S ^ t , m v t , m ∈ [ 1 , M ] (6)

[0075] In the formula, represents the aggregated features of the mth stage; Represents the value obtained by using the Softmax function in the time dimension Normalized coefficients for feature aggregation between values ​​V, m s is the starting frame number of the mth stage, m e is the cutoff frame number of the mth stage. The attention coefficient of each action stage is obtained by the attention unit in the multi-stage attention decoder with the feature prototype of the action stage:

[0076] (7)

[0077] Among them, a m represents the attention weight coefficient of the mth action stage, represents the attention unit operation, P m represents the query sequence of the mth action stage, that is, P D (m) or P S (m).

[0078] Then, using the obtained aggregated features of the mth stage and the attention weight coefficient a m , perform weighted combination of the dynamic or static features of the stage to obtain the adjusted dynamic and static features of each action stage:

[0079] (8)

[0080] Finally, the dynamic and static features of each action stage of the complete video are processed by a feedforward neural network to obtain the final dynamic flow feature F D and the static flow characteristics F S .

[0081] In this embodiment, the score prediction module adopts a regression model. The score regression module consists of three fully connected layers and a Sigmod activation function. Its input is the dynamic flow feature F fused in a cascade manner. D and the static flow characteristics F S .

[0082] The final score for the quality of the pull-up movement is expressed as:

[0083] s = S i g m o i d ( F c [ F D ; F S ] ) (9)

[0084] In the formula, represents a fully connected layer.

[0085] A score threshold is preset, and the pull-up action quality result is given according to the difference between the predicted score and the preset score threshold.

[0086] The last layer of the score regression module uses the Sigmoid activation function to normalize the predicted score to the range of [0, 1]. In this way, no matter what the original score is, the normalized score is between [0, 1], making the model prediction results more convergent and universal.

[0087] Before training the Transformer-based pull-up action quality assessment model, a dataset for training the Transformer-based pull-up action quality assessment model is first built.

[0088] The data set includes multiple complete action video frames, and each complete action video frame contains several continuous frames; the data set is preprocessed as follows: a video stream of a complete action is subjected to frame extraction, the length of each video is about 3 seconds, the frame rate FPS is 30, and this embodiment extracts 16 frames from a complete action video with a frame size of 512×512.

[0089] Each video is annotated by experts with score labels and three stages: torso rising, chin over the bar, and torso lowering.

[0090] Then, using the data set, we train a Transformer-based pull-up action quality assessment model. The training process is as follows:

[0091] The experiment uses the Adam optimizer, with a batch size of 32 for training, a learning rate of 0.001, and a training cycle (epoch) of 250. Through the back-propagation algorithm, the model parameters are adjusted to minimize the difference between the predicted score and the true score, thereby guiding the model to learn to more accurately capture the characteristics of the pull-up action.

[0092] In addition, the present invention uses the Spearman correlation coefficient as an evaluation index to evaluate the performance of the model in the task, ensuring that the model trained by the present invention can effectively reflect the quality of the pull-up action.

[0093] The self-built pull-up dataset was fed into the pull-up action quality assessment model for training. The batch size was set to 32, the learning rate was 0.001, and when the training reached 250 rounds, the Spearman correlation coefficient was the highest, which means the prediction result was the best.

[0094] After the model is trained, the Transformer-based pull-up action quality assessment model gives a specific score for the input pull-up video and gives the action quality assessment result.

[0095] The preset score threshold is set to 0.45 points, for example. A pull-up action is predicted to score 0.4 points through the Transformer-based pull-up action quality assessment model. The model will compare this predicted score with the preset score threshold of 0.45 points. Since 0.4 points is lower than the preset threshold of 0.45, the model will determine that this pull-up action does not meet the perfect standard. If the model predicts that the score of another pull-up action is 0.48 points, then this score is higher than the preset threshold of 0.45 points, and the model will determine that the quality of this pull-up action has reached the perfect standard.

[0096] In addition, in order to verify the effectiveness and practicality of the method proposed in this invention, the laboratory's previously released dataset SDUST-PUT1 and the recently collected self-built dataset SDUST-PUT2 were used for testing.

[0097] The SDUST-PUT1 dataset contains 2,000 jpg images of size 512 × 512, of which 1,737 are pull-up actions taken by volunteers and 263 are pull-up actions extracted from online videos.

[0098] From these images, 16 consecutive frames with the same label as SDUST-PUT2 are selected to form a complete action video frame, and a total of 1,600 images are selected to form 100 complete action video frames. The SDUST-PUT2 dataset contains 170 complete pull-up action videos shot by myself, with a frame rate of 30 FPS and a length of about 3 seconds for each video.

[0099] Each video is annotated by experts with score labels and three stages: torso rising, chin over the bar, and torso lowering.

[0100] The present invention constructs a pull-up dataset, and can customize the data collection and annotation process according to actual target requirements to obtain more complete, detailed and representative data. Such a dataset will be more universal and can make a more accurate evaluation of pull-up actions in various actual scenarios. It also fills the gap in the current pull-up dataset.

[0101] After the model training is completed, in order to verify the results of the model prediction, the evaluation indicator uses the universal Spearman rank correlation coefficient to reflect the correlation between the algorithm prediction score and the true label score, and the test set is used to evaluate the performance of the model.

[0102] The definition is as follows:

[0103] (10)

[0104] In the formula, s represents the action quality score, and the subscript represents the sample number, the subscript P represents the algorithm prediction, and the subscript T represents the true label; the value range of ρ is [-1, 1], and the higher the value, the higher the correlation between the prediction score and the true label score;

[0105] represents the prediction score of the i-th video, represents the average score of all predicted pull-up movements, represents the true score of the i-th video, Represents the true average score for all pull-up movements.

[0106] All algorithms in this paper are implemented using Python 3.8 and Pytorch 1.10.2.

[0107] The number of pull-up action stages M is set to 3, and the number of frames T is 16. For the SDUST-PUT1 dataset, among the 100 complete action video frames, 70 complete action video frames are selected as the training set, and 30 complete action video frames are selected as the test set. For the SDUST-PUT2 dataset, 140 videos are selected as the training set, and 30 are selected as the test set.

[0108] During the training process, the Adam optimizer is used for back propagation. The batch size is set to 32, the learning rate is 0.001, and the epoch is 250. The hardware configuration of the running environment is GeForce RTX 2080Ti 11G Nvidia GPU.

[0109] In order to better illustrate the technical effect of the present invention, four comparison methods are selected for comparative experiments. The four comparison methods are: C3D-SVR, TSA-Net, ACTION-NET, GDLT, DS-MSAT. The experimental results on the SDUST-PUT1 and SDUST-PUT2 datasets, as well as the comparison results with the current mainstream AQA algorithms are shown in Table 2.

[0110] For the above four compared methods, except for modifying the feature extraction part and its final scoring module to be suitable for pull-up movement quality assessment, no other changes were made.

[0111] It can be seen that DS-MSAT has obvious advantages in both score prediction accuracy and calculation time.

[0112] Table 2 Prediction results of pull-up movement quality scores

[0113]

[0114] Since the C3D-SVR model uses the classic C3D network to extract features, the high-dimensional feature space obtained by it shows obvious sparsity in the pull-up dataset, resulting in the worst score prediction effect despite the high computational cost. TSA-Net, ACTION-NET and GDLT have comparable performance on the SDUST-PUT1 dataset, while TSA-Net performs slightly worse on the SDUST-PUT2 dataset. The feature extraction parts of the three algorithms are basically similar, all using I3D extractors, while ACTION-NET further uses Resnet to extract features. The main reason for the performance difference among the three is the adjustment of the network architecture other than feature extraction. Based on video features, TSA-Net introduces optical flow auxiliary information to enhance the spatiotemporal modeling ability of video actions. ACTION-NET uses a two-stream network based on the GCN framework to extract dynamic and static features. The GDLT model uses the advantages of the Transformer structure in processing sequence data to effectively integrate the features extracted by I3D into the Transformer architecture. The computational costs of the three models on the SDUST-PUT2 dataset are also basically comparable, at 357, 364, and 356 seconds respectively. DS-MSAT draws on the ideas of ACTION-NET in terms of structure and feature extraction strategy. The difference is that DS-MSAT can more effectively capture the features and details of the pull-up video action by introducing a multi-stage attention decoding module, and effectively model the scoring process. It can be seen that the DS-MSAT model proposed in the present invention reaches ρ of 0.896 for the SDUST-PUT1 data set, and ρ reaches 0.936 for the SDUST-PUT2 data set, which are significantly higher than the compared algorithms. Since the SDUST-PUT1 data set is a video frame sequence selected from 2000 pictures built by the laboratory in the early stage, the fluency of its action and the temporal sequence between adjacent frames are slightly poor. Therefore, the difference in the correlation coefficients on the two data sets is mainly due to the different temporal distribution characteristics of the data sets.

[0115] In addition, in order to evaluate the role of dynamic flow branches and static flow branches in the DS-MSAT model, corresponding ablation experiments were performed on the SDUST-PUT1 and SDUST-PUT2 datasets, as shown in Table 3.

[0116] Table 3 Ablation experiment results

[0117]

[0118] It can be seen that the correlation coefficients of the model using a single-branch dynamic flow structure on the two data sets are 0.854 and 0.872 respectively. The correlation coefficients of the model using a single-branch static flow structure on the two data sets are 0.874 and 0.897 respectively. The dynamic flow branch is conducive to capturing the continuity of actions in video sequences, but it is easy to ignore irregular actions. The static flow branch is more conducive to identifying the spatial information of posture details and morphological norms in key frames, providing supplementary information for the dynamic flow branch. Therefore, the correlation coefficients of the dual-stream branch model reached 0.882 and 0.906 respectively. Since different stages of the pull-up action contribute differently to the final evaluation results, the present invention introduces the MSAD module on the basis of the dual-stream branch, and the correlation coefficient on the SDUST-PUT1 data set is increased from 0.882 to 0.896, and the correlation coefficient on the SDUST-PUT2 data set is increased from 0.906 to 0.936, reflecting that the segmented weighted quality scoring strategy of the MSAD module has a significant improvement effect on the accuracy of prediction.

[0119] The pull-up action quality assessment method proposed in the present invention divides the pull-up action into three key stages: body rise, head and chin passing the bar, and body descent, according to the action characteristics of the pull-up. The present invention trains the data set based on the pull-up action quality assessment model of Transformer, accurately captures each stage of the pull-up action, and then accurately scores the action quality. By constructing a model specifically for scoring the pull-up action, the problems of overfitting and underfitting of the model that may exist when the general model handles the task can be avoided, so that the generalization ability and prediction performance of the model are significantly improved. The present invention is closer to actual needs, can effectively reduce resource consumption, and improve efficiency. The method of the present invention is to evaluate the action quality by labeling the self-built pull-up data set and sending it to the built pull-up action quality assessment model based on Transformer, so as to construct a pull-up-specific automatic assessment model, which can greatly reduce manpower and material resources.

[0120] Example 2

[0121] This embodiment 2 describes a computer device, which includes a memory and one or more processors. The memory stores executable code. When the processor executes the executable code, it is used to implement the steps of the pull-up action quality assessment method in the above embodiment 1.

[0122] In this embodiment, the computer device is any device or apparatus with data processing capability, which will not be described in detail here.

[0123] Example 3

[0124] This embodiment 3 describes a computer-readable storage medium on which a program is stored. When the program is executed by a processor, it is used to implement the steps of the pull-up action quality assessment method in the above-mentioned embodiment 1.

[0125] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0126] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A method for evaluating the quality of pull-up movements, characterized in that: The steps include: Step 1. According to the characteristics of the pull-up movement, design a phased action normative evaluation standard and scoring method; divide the pull-up video action sequence into three key stages: body rising, head and chin passing the bar, and body descending; Step 2. Build a Transformer-based pull-up action quality assessment model, which includes a feature extraction network with a dual-branch structure, a feature fusion network, and a score regression module; The feature extraction network with a dual-branch structure includes a dynamic flow feature extraction network branch and a static flow feature extraction network branch, and is used to extract dynamic flow output features and static flow output features respectively; The dynamic flow output features and the static flow output features are fused in the feature fusion network, and the fused results are sent to the score regression module to obtain the predicted score, and the pull-up action quality evaluation result is given according to the predicted score; The dynamic flow feature extraction network branch includes a dynamic flow feature extractor, a context-aware encoder, and a multi-stage attention decoder; the processing flow of the input video frame in the dynamic flow feature extraction network branch is as follows: The dynamic flow feature extractor first extracts dynamic information, that is, captures and analyzes the motion changes of adjacent frames in the video frame sequence, identifies the posture and position changes between each frame of the human body during the pull-up process, learns the timing information of the three stages of torso rising, head and chin passing the bar, and torso descending, and obtains dynamic flow features. The dynamic flow features are input into the context-aware encoder, and the correlation features between frames are calculated to obtain dynamic flow enhancement features. The dynamic flow enhancement features and the feature prototypes of the three action stages of dynamic flow are simultaneously input into the multi-stage attention decoder, and the complete video of the pull-up action is divided into key stages, and then the corresponding weight coefficient is assigned to each key stage and weighted processing is performed to obtain the dynamic flow output features; The static stream feature extraction network branch includes a static stream feature extractor, a context-aware encoder, and a multi-stage attention decoder. The processing flow of the input video frame in the static stream feature extraction network branch is as follows: The static flow feature extractor first extracts posture information, i.e., the position, posture, and spatial relationship of each part of the human body in each frame, to obtain static flow features. The static flow features are input into the context-aware encoder, and the correlation features between frames are calculated to obtain static flow enhancement features. The static flow enhancement features and the feature prototypes of the three action stages of the static flow are simultaneously input into the multi-stage attention decoder. The complete video of the pull-up action is divided into key stages, and then the corresponding weight coefficient is assigned to each key stage and weighted processing is performed to obtain the static flow output features. The processing process of the context-aware encoder is as follows: First, for the input i-th video, the initial features of its T-frame image are weighted and aggregated using the self-attention mechanism to obtain context information; then, the feature vector of the context information is added and normalized to the initial feature vector and then passed to the feedforward neural network for fusion; finally, the enhanced features are obtained; The formula for calculating the static flow features using the context-aware encoder is as follows: In the formula, A feature vector representing context information, represents the initial feature vector of the static flow extracted by the static flow feature extractor, norm(·) represents layer normalization, F represents the video features after addition and normalization, represents the enhanced features obtained after processing by the context-aware encoder in the static stream; DNN θx Denote the parameter as θ x Feedforward neural network; The formula for calculating the dynamic flow features using the context-aware encoder is as follows: In the formula, A feature vector representing context information, represents the initial feature vector of the dynamic flow extracted by the dynamic flow feature extractor, norm(·) represents layer normalization, F represents the video features after addition and normalization, Represents the enhanced features obtained after processing by the context-aware encoder in the dynamic flow; DNN θx Denote the parameter as θ x Feedforward neural network.

2. The pull-up movement quality assessment method according to claim 1, characterized in that: The dynamic flow feature extractor adopts an expanded three-dimensional convolutional neural network I3D; The static flow feature extractor adopts the Resnet-50 network.

3. The pull-up movement quality assessment method according to claim 1, characterized in that: The multi-stage attention decoder is an improvement on the traditional Transformer decoder. By proposing an attention unit, a multi-stage attention network is formed. The attention unit consists of two fully connected layers, one rectified linear unit and one activation function to obtain the weight coefficients of different action stages; The specific processing flow of the multi-stage attention decoder is as follows: First, masked multi-head attention is used to mine the relationship between the feature prototypes of the M action stages, and the updated feature prototype is recorded as and In the self-attention mechanism, each element of the input sequence is converted into three different vectors: query Q, key K, and value V; Then, multi-head attention is used to mine the enhanced features f′(t) and The relationship between and Perform a linear projection to transform it into the corresponding query vector: In the formula, q D (m) represents the feature prototype of the mth action stage in the dynamic flow, q S (m) represents the feature prototype of the mth action stage in the static flow; W q is the projection parameter matrix of the dynamic flow query vector, W q ′ is the projection parameter matrix of the static flow query vector; f′(t) is the enhanced feature generated in the dynamic flow or static flow, that is, or Represents the dynamic or static features of the tth frame of the input video; The key K and value V are transformed by f′(t) through different linear mappings: In the formula, k t represents the K vector of the tth frame, v t represents the V vector of the tth frame; W k is the projection parameter matrix of the key vector, W v is the projection parameter matrix of the value vector; the semantic relevance between the feature prototype of the mth action stage and the tth frame of the video is obtained by the dot product similarity S between the corresponding query vector q and the key k t,m To measure: In the formula, Represents the scale factor, divided by To prevent gradient explosion; That is or represents the query vector of the mth action stage in a dynamic or static flow, and the symbol Γ represents the transpose of the vector; The aggregated features of the m-th action stage are normalized by the coefficients The linear combination with the value V of the action phase frame is obtained: In the formula, f agg (m) represents the aggregation characteristics of the mth stage; Represents S obtained by using the Softmax function in the time dimension t,m Normalized coefficients for feature aggregation between values ​​V, m s is the starting frame number of the mth stage, m e is the cutoff frame number of the mth stage; the attention coefficient of each action stage is obtained by the attention unit in the multi-stage attention decoder with the feature prototype of the action stage: Among them, a m represents the attention weight coefficient of the mth action stage, represents the attention unit operation, P m represents the query sequence of the mth action stage, that is, P D (m) or P S (m); P D (m) and P S (m) represents the feature prototype of the mth action stage in the dynamic flow and static flow branches, i.e., the query feature, m∈[1,M]; Then, using the obtained aggregated feature f of the mth stage agg (m) and attention weight coefficient a m , perform weighted combination of the dynamic or static features of the stage to obtain the adjusted dynamic and static features of each action stage: Finally, the dynamic and static features of each action stage of the complete video are processed by a feedforward neural network to obtain the final dynamic flow feature F D and the static flow characteristics F S .

4. The pull-up movement quality assessment method according to claim 1, characterized in that: In step 2, the score regression module consists of three fully connected layers and a Sigmod activation function, and its input is the dynamic flow feature F fused in a cascade manner. D and the static flow characteristics F S ; The final predicted score for the quality of pull-up movement performance is expressed as: s=Sigmoid(F c [F D ;F S ]) (9) Among them, the Sigmod activation function normalizes the predicted score to the range of [0,1], F c represents a fully connected layer; A score threshold is preset, and the pull-up action quality result is given according to the difference between the predicted score and the preset score threshold.

5. The pull-up movement quality assessment method according to claim 1, characterized in that: The step 1 is specifically as follows: The entire pull-up action video is divided into three key stages, namely, the torso rising stage, the chin-over-the-bar stage, and the torso descending stage, and weight coefficients are assigned to the corresponding key stages; according to this ratio, if the total score of the action quality is 5 points, the full score of the torso rising stage is 2 points, the full score of the chin-over-the-bar stage is 1.5 points, and the full score of the torso descending stage is 1.5 points; During the stage of rising the body, if the arms are not straightened, the body swings or the legs are bent at the start, 1.0, 0.2 and 0.2 points will be deducted respectively; during the stage of chin-over-bar, if the legs are bent and the chin is not over the bar, 0.2 points and 1.0 points will be deducted respectively; during the stage of descending the body, if the legs are close together and the arms are not straightened when lowering, 0.2 points and 1.0 points will be deducted respectively.

6. The pull-up movement quality assessment method according to claim 1, characterized in that: In the step 2, before training the Transformer-based pull-up action quality assessment model, firstly, a data set for training the Transformer-based pull-up action quality assessment model is constructed; The data set includes multiple complete action video frames, and each complete action video frame contains several continuous frames; The dataset is preprocessed as follows: a video stream of a complete action is subjected to frame extraction, each video is 3 seconds long, the frame rate FPS is 30, 16 frames are extracted from a complete action video, and the frame size is 512×512; Each video is annotated with a score label and three stages: torso rising, chin over the bar, and torso lowering.

7. A computer device comprising a memory and one or more processors; an executable code is stored in the memory; characterized in that: When the processor executes the executable code, it is used to implement the pull-up action quality assessment method described in any one of claims 1 to 5.

8. A computer-readable storage medium having a program stored thereon; characterized in that: When the program is executed by a processor, it is used to implement the pull-up action quality assessment method described in any one of claims 1 to 5.

Images