Action spatio-temporal perception quality evaluation method based on text rule prompting learning

Abstract

The application discloses a motion spatiotemporal perception quality evaluation method based on text rule prompt learning, and mainly solves the problems of great evaluation limitation and low accuracy rate in the prior art.The implementation scheme is as follows: action videos of different fitness methods are collected and labeled according to requirements to divide training set and test set; based on the existing CLIP model, a motion spatiotemporal perception quality evaluation network based on text rule prompt learning is constructed; time process clues of action rules are mined, and video action features and rule text features which have a direct influence on evaluation results are extracted; the video action features and the rule text features are subjected to multi-modal fusion; the training set is used to train the motion spatiotemporal perception quality evaluation network based on text rule prompt learning; test data is input into the trained quality evaluation network to give an action score and a result of whether the action is standard.The application reduces the cost of action quality evaluation, improves the accuracy of evaluation, and can be used for fine-grained sports fitness action quality evaluation.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of deep learning and video understanding, and particularly relates to a video action quality evaluation method, which can be used for fine-grained sports fitness action quality evaluation. BACKGROUND

[0002] Action quality evaluation is a research on the quantification of human performance level when performing actions, involving multiple fields such as sports, medicine and teaching. In the field of sports, the performance evaluation of athletes usually relies on experienced coaches or experts in the field. However, this approach is limited by the time of experts and the subjectivity of evaluation, and it is difficult to provide instant, accurate and consistent feedback. In the medical field, when evaluating the rehabilitation process of patients, the precision of surgical operations and the effectiveness of physical therapy, it is necessary to accurately measure the quality of actions. In traditional medical practice, doctors or therapists usually rely on their professional knowledge and experience, which is inefficient, time-consuming and labor-intensive, and is easily affected by subjective factors. Therefore, it is necessary to establish a computer vision model to automatically evaluate the quality of human actions to accurately capture and analyze subtle action differences and provide effective feedback and guidance for humans.

[0003] Action quality evaluation has gradually received widespread attention in recent years, especially in competitive sports such as diving, gymnastics, figure skating and other high-precision and expressive projects. Subtle action differences in these projects can cause significant changes in scores, so it is necessary to accurately evaluate the quality of athletes' actions. Current research uses rich, easily accessible and authoritative datasets in competitive sports, mainly focusing on using deep learning regression tasks to predict action scores. In contrast, in the field of fitness exercises, especially in daily environments, there is less relevant research. Due to the lack of clear scoring criteria, it is currently challenging to quantitatively score the quality of fitness actions. In daily fitness action evaluation, it is usually necessary to determine whether the action is standardized according to the rules, and this task is essentially a multi-label binary classification task. However, in the field of action quality evaluation, especially in the fine classification of fitness exercises, current research is still insufficient.

[0004] The patent document with publication number CN113663312A discloses a method for evaluating the quality of instrument-free fitness movements based on micro-inertia. The method uses two-step classification of state and action. Firstly, it uses three 6-axis micro-inertia sensors to identify human movement states, then calls the Elman Kalman trajectory estimation model to predict the movement trajectory of each sensor node, and then uses the trajectory signal and the collected signal for the second step of fitness movement classification. After classification, the generated movement sequence is compared and analyzed with the sequences in the standard movement sequence library to evaluate the standard and stability of the whole body and local movements, and the quality evaluation result of the movement is given. However, this method uses 6-axis micro-inertia sensors, which is relatively high in cost and low in evaluation accuracy, and also limits the recognizable action categories, which cannot be used in all scenarios.

[0005] The patent document with publication number CN113255489A discloses a multi-modal diving competition intelligent evaluation method based on marker distribution learning. The method is based on a dual-mode inflated 3D convolution network (I3D) that adds a light flow to extract the spatiotemporal features of RGB and Flow modal diving videos and forms global features by fusion, which are input into the fully connected layer. The marker distribution learning LDL is used to output the probability distribution of the average score of all judges, and the maximum probability score is selected to multiply the difficulty coefficient to obtain the final diving score. However, this method can only evaluate the quality of diving, and since it uses light flow extraction, the network implementation process is complex and the running speed is slow.

[0006] In addition, the above-mentioned existing methods only focus on visual information during fitness exercise, do not analyze and evaluate spatial features related to evaluation rules, and ignore the role of time process clues in action rule texts at key moments of movement. Therefore, subtle differences in movements cannot be found during the movement process, resulting in poor evaluation accuracy. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a method for evaluating the quality of action spatiotemporal perception based on text rule prompt learning. According to the time process clues of the action rule text and the time correlation between the action process and the score, the key moments that have a decisive influence on the action score are determined, and the accuracy of action evaluation is improved using these key moments.

[0008] To achieve the above-mentioned purpose, the implementation scheme of the present application includes the following steps:

[0009] (1) Collect action videos of different fitness methods, label them according to requirements, and divide the training set and test set according to different proportions according to the category of the action;

[0010] (2) Based on the existing CLIP model, a motion spatio-temporal perception quality evaluation network based on text rule prompt learning is constructed:

[0011] (2a) A fine-grained video spatio-temporal encoder module composed of a video encoder module and a 3D transpose convolution module is established to extract fine-grained basic video features of the video;

[0012] (2b) A time course clue mining module based on the existing CLIP image encoder and CLIP text encoder is established to generate time attention Sim n ;

[0013] (2c) A multi-modal rule space perception fusion module based on visual and rule text features is established to extract multi-modal fusion features that directly affect the evaluation results by utilizing visual and text information;

[0014] (2d) The modules established in steps (2a) to (2c) are sequentially connected, and the multi-modal fusion features are connected to a plurality of multilayer perceptrons with the same number of fitness exercise rules to form a motion spatio-temporal perception quality evaluation network based on text rule prompt learning, and the existing BCEWithLogitsLoss loss function is used as the loss function Loss of the network;

[0015] (3) The training set data is input into the above motion spatio-temporal perception quality evaluation network, and the AdamW optimization algorithm is used to train it until the network loss function is in a convergent state, and a trained quality evaluation network is obtained:

[0016] (4) The test set data is input into the trained motion spatio-temporal perception quality evaluation network to obtain the estimated results corresponding to the test data.

[0017] Compared with the prior art, the present application has the following advantages:

[0018] Firstly, the learning ability of the network for action details is increased.

[0019] The present application can help the network better understand the subtle differences of action details in the fitness exercise process by mining the time course clue information of the action rules through the establishment of the time course clue mining module, thereby improving the evaluation accuracy. At the same time, the present application can further mine visual and rule text features and extract action detail features that directly affect the evaluation results through the establishment of the multi-modal rule space perception fusion module, so that the action detail features are more easily learned by the network.

[0020] Secondly, the accuracy and precision of the action quality evaluation are improved.

[0021] The application introduces a corresponding number of 3D transpose convolution and global 3D maximum pooling operations for each rule of the fitness action to obtain fine-grained video features, and combines the text space prompt to further mine the spatio-temporal information of the obtained video features, and introduces a graph convolution network to alleviate the problem caused by label imbalance, thereby improving the accuracy and precision of the evaluation quality.

[0022] Thirdly, the use cost is reduced.

[0023] The application performs spatio-temporal quality evaluation of actions based on common RGB videos, so the required hardware equipment cost is low and is suitable for large-scale popularization and application. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 is the implementation flowchart of the application;

[0025] Figure 2 is the action spatio-temporal perception quality evaluation network structure diagram in the application;

[0026] Figure 3 is Figure 2 the rule relationship mining structure diagram. DETAILED DESCRIPTION

[0027] In order for those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, other embodiments obtained by those skilled in the art without creative labor should be within the protection scope of the present application.

[0028] With reference to Figure 1 , the implementation steps of the present example are as follows:

[0029] Step 1. According to the requirements, the collected different fitness action videos are labeled, and the training set and the test set are divided according to the different proportions according to the category of the action.

[0030] (1.1) Use the Azure Kinect device with a video frame rate of 30fps and a resolution of 720P to collect videos of different fitness actions;

[0031] (1.2) According to the following cross-validation labeling strategy, the collected videos are labeled for each action starting frame, ending frame and action standard degree:

[0032] (1.2.1) Two volunteers labeled all video samples respectively, that is, accurately divided and labeled the start frame and end frame of each repetitive action in the video sample, and gave a standard score for each repetitive action according to the fitness action rules.

[0033] (1.2.2) The third volunteer cross-validates the annotation results of the first two volunteers. If a large difference is found between the two annotation results, the third volunteer needs to re-evaluate the video and modify its annotation.

[0034] (1.3) Divide the collected videos into training and testing sets, taking the pull-up exercise as an example:

[0035] Actions in the captured videos that involve significant body displacement, with a swing amplitude exceeding 30° and an error rate exceeding 40%, are divided into training and testing sets in a 4:1 ratio.

[0036] Actions in the captured videos with small body displacement, swing amplitude not exceeding 30°, and error rate less than 40% were divided into training and test sets in a 1:1 ratio.

[0037] Step 2. Construct an action spatiotemporal perception quality assessment network based on text rule prompting learning.

[0038] Reference Figure 2 The specific implementation of this step is as follows:

[0039] (2.1) Extract fine-grained basic video features from the video using a fine-grained video spatiotemporal encoder:

[0040] (2.1.1) Extract video features F from the input video V using the video encoder module. v Its dimensions are Where V∈R T×H×W×3 T is the number of video frames, H and W represent the height and width of the video respectively, and D is the number of channels;

[0041] (2.1.2) For video features F v The time channel is extended by upsampling from T / 2 to T, and then fine-grained basic video features F are obtained after passing through a 3D transposed convolution module and a global 3D max pooling operation. rule Its dimensions are M×T×D, where M is the number of fitness movement rules;

[0042] (2.2) Using the temporal process cue mining module, the correlation between video frames and action process cue text is used to generate temporal attention Sim. n :

[0043] (2.2.1) Input the video V into the CLIP image encoder to generate CLIP image visual features F with dimensions T×D. cv Simultaneously, based on the M fitness movement rule texts, a fine-grained text description T containing a subset of all rule texts is constructed. d It is then fed into the CLIP text encoder to generate a 2-dimensional text encoder. M The CLIP text feature F of ×D ct ;

[0044] (2.2.2) Along the time dimension, CLIP text features F ct Visual features F of CLIP images cv The components are concatenated to generate a dimension of (T+2). M The splicing feature F of )×D c The concatenated features are adaptively fused using a temporal Transformer to generate CLIP temporal image features F with dimensions T×D. image ;

[0045] (2.2.3) Generate clue text T using the time period information corresponding to the action and the rule. n This data is then fed into the CLIP text encoder to generate learnable cue features F of dimension N×D. text , where N is the number of clues;

[0046] (2.2.4) Using the similarity calculation formula to measure CLIP temporal image features F image and learnable cue features F text The similarity between the nth clue text sentence in frame t is calculated using the following formula:

[0047]

[0048] In the formula, t∈[1,T], n∈[1,N], and the superscript ' indicates transpose;

[0049] (2.2.5) Calculate the similarity of all video frames in the input video to obtain T similarity values, and then form a similarity matrix along the time dimension to obtain the action-time process attention Sim corresponding to each cue text sentence. n :

[0050]

[0051] In the formula, Sim n ∈R T , which represents the correlation between all video frames in the input video and the temporal process information contained in the nth clue text sentence;

[0052] (2.3) The multimodal rule space perception fusion module is used to extract multimodal fusion features that have a direct impact on the evaluation results using visual and textual information:

[0053] (2.3.1) Multiply the time process attention Sim n generated in step (2.2.5) with the CLIP time sequence image features F image generated in step (2.2.2) to obtain CLIP video features F clip directly related to the action process:

[0054]

[0055] where F clip ∈R M×T×D , M is the number of fitness action rules, T is the number of input video frames, and D is the number of channels; F clip m = F image ·(1-Y m,n )+F image ·Y m,n ·Sim n represents the CLIP video features of the mth rule related to the action process, m ∈ {1, 2, …, M}, and M rules correspond to M video features F clip (1)～(M) ; Y m,n ∈ {0, 1} indicates whether the action rule m includes the corresponding time process attention: if it is 1, it means that the action rule has corresponding time process attention; otherwise, it has no corresponding time process attention.

[0056] (2.3.2) Multiply the fine-grained basic video features F rule generated by the fine-grained video encoder module in step (2.1.2) with the time process attention Sim n generated in step (2.2.5) to obtain fine-grained video features F clip that are time-aligned with the CLIP video features F align :

[0057] F align = {F align (1) ,F align (2) ,...,F align (m) ,...,F align (M)}

[0058] where F align ∈R M×T×D , F alignm = F rule m · (1 - Y m,n ) + F rule m · Y m,n · Sim n denotes the m-th rule corresponding to the fine-grained video feature;

[0059] (2.3.3) The evaluation rule prompt of the fitness action is generated around the key evaluation points of the action, and is input to the CLIP text encoder to generate a learnable rule feature F of dimension M x D eva , which is then copied and expanded to match the dimension of the CLIP video feature F clip , to obtain a text feature F eva ' of dimension M x T x D;

[0060] (2.3.4) The CLIP video feature F clip and the text feature F eva ' are spliced in the feature D dimension, and a linear projection with weight W q ∈ R 2D×D is used to generate a multi-modal query Q of dimension M x T x D: Q = Concat (F clip , F eva ') W q ;

[0061] (2.3.5) The multi-modal query Q is fused with the fine-grained video feature F align to obtain the fusion feature H:

[0062] First, the time aggregation operation is performed on Q and F align to generate more representative query feature Q' and fine-grained video feature F align ':

[0063] Q' = AggregateT (Q)

[0064] F align ' = AggregateT (F align )

[0065] Then, Q' and F align ' are added and then nonlinearly transformed by the tanh activation function to obtain the fusion feature H of dimension M x D:

[0066] H = tanh (Q' + F align ');

[0067] (2.3.6) The attention score is calculated by a linear transformation layer on the fusion feature H, and is normalized by a Softmax function to obtain the attention weight A with a dimension of MxD:

[0068] A = Softmax(HW a )

[0069] In the formula, W a is a linear projection with weights R D×D ;

[0070] (2.3.7) The attention weight A is used to further extract the feature, and the multi-modal fusion feature F directly affecting the evaluation result is extracted:

[0071] First, the attention weight A is copied in the time dimension to obtain the attention weight matrix A' with a dimension of MxTxD, and then it is multiplied with the multi-modal query feature Q and the fine-grained video feature F align to obtain Q" and F align ":

[0072] Q" = A' 0 Q

[0073] F align " = A' 0 F align

[0074] Then, after the Q" and F align " are spliced in the feature channel, a linear projection is performed to generate the final multi-modal fusion feature F with a dimension of MxTxD:

[0075] F = Concat(Q", F align ") W f

[0076] In the formula, W f is a linear projection with weights R 2D×D ;

[0077] (2.3.8) The graph convolution network is introduced to extract information from the modal fusion feature F, and the relationship between each rule is mined to obtain the relationship vector Vector 1-M corresponding to the fitness action rule:

[0078] Referring to Figure 3 , the specific implementation of this step is as follows:

[0079] The multi-modal fusion feature F is aggregated in the time dimension to obtain the aggregated fusion feature F' with a dimension of MxD, and then the global representation F g ∈ R D is obtained after the global average pooling and a convolution layer.

[0080] The global representation Fg Extended M-dimensional post-series aggregation fusion feature F' to get intermediate feature F" ∈ R 2D×M , and then linearly projected with weight W A ∈ R D×D through the Sigmoid activation function to obtain the dynamic correlation matrix A reflecting the specific rule dependency: d = δ(W A F"), where δ(·) is the Sigmoid activation function.

[0081] The aggregation fusion feature F' is input into the dynamic GCN network, and a static correlation matrix A s is set to capture the coarse-grained category dependency as the basis of global rule relationship, and then it is superimposed with the dynamic correlation matrix A d , and then the state update weight matrix and LeakyReLU activation function are used to output the relationship matrix P with dimension M × D:

[0082] P = f((A s +A d )F'W d )

[0083] where f(·) is the LeakyReLU activation function, W d ∈ R D×D is the state update weight matrix.

[0084] The relationship matrix P is then passed through the multi-layer perceptron and the Sigmoid activation function to obtain the relationship vector set Vector 1-M corresponding to each piece of fitness action rule:

[0085] Vector 1-M = {Vector1, Vector2,..., Vector m ,..., Vector M}

[0086] where M is the number of fitness action rules, and the dimension of each relationship vector is M × 1.

[0087] (2.4) Connect the multi-modal fusion feature to the multi-layer perceptron with the same number of fitness exercise rules to obtain the classification result of each rule:

[0088] The multi-modal fusion feature F is input into the multi-layer perceptron, and the multi-modal fusion feature F is input into M multi-layer perceptrons according to the M rules of the fitness action in the video, and each multi-layer perceptron is responsible for the evaluation of one rule.

[0089] The output result obtained after passing the multi-modal fusion feature F through each multi-layer perceptron is multiplied by the rule relationship vector {Vector1, Vector2,..., Vector M} corresponding to the multi-layer perceptron, to obtain the classification result of each rule;

[0090] (2.5) The modules established in steps (2.1)-(2.4) are sequentially connected to form an action space-time perception quality evaluation network based on text rule prompt learning;

[0091] (2.6) The existing Sigmoid activation function and binary cross-entropy loss are combined to form a loss function Loss of the action space-time perception quality evaluation network:

[0092]

[0093] Where y is the true value, and the prediction result.

[0094] Step 3. Input the training set data into the action space-time perception quality evaluation network and train it using the AdamW optimization algorithm.

[0095] (3.1) Given a video, input it into the fine-grained video encoder module in the network to generate fine-grained basic video features F rule ;

[0096] (3.2) The video is simultaneously input into the time course clue mining module in the network to generate CLIP time sequence image features F image and time course clue features F text , and similarity calculation is performed between the two to generate time course attention Sim;

[0097] (3.3) The time course attention Sim is multiplied with the CLIP time sequence image features F image and the fine-grained basic video features F rule to generate CLIP video features F clip and fine-grained video features F align that are time-aligned with the CLIP video features F clip ;

[0098] (3.4) Integrate the action rule text description, CLIP video features F clip and fine-grained video features F align , pass them through the multi-modal rule space perception fusion module in the network to generate multi-modal fusion features F directly related to the scoring rules, and perform rule relationship mining to generate relationship vectors Vector 1-M ;

[0099] (3.5) Multiply the multi-modal fusion features F and the relationship vectors Vector1-M The classification result is obtained through the multilayer perceptron post-computation in the network;

[0100] (3.6) The error between the classification result and the true value is calculated by using the loss function Loss, the error information is back-propagated, and the weight parameters of the network are adjusted by minimizing the Loss;

[0101] (3.7) Repeat steps (3.4) to (3.6) until the training times reach the set threshold or the value of the loss function converges, and obtain the trained action spatiotemporal perception quality evaluation network.

[0102] Step 4. Input the test data set into the trained action spatiotemporal perception quality evaluation network to obtain the evaluation result corresponding to the test data:

[0103] (4.1) Randomly take b video samples from the test set data, and pre-set M action rule texts according to the video action category of each video, input them into the network together with the video sample, output M action rule quality evaluation results corresponding to the M action rule texts, and each evaluation result can only be standard or not standard;

[0104] (4.2) Output the final action evaluation quality of the test video according to the action rule quality evaluation result:

[0105] When the M action rule quality evaluation results are all standard, the quality evaluation result of the video action is output as standard;

[0106] On the contrary, if there is a non-standard result in the M action rule quality evaluation results, it is judged that the quality evaluation result of the video action is that there is a violation.

[0107] The effects of the present application can be further illustrated by the following simulation.

[0108] I. Simulation data

[0109] Two data sets are used, namely the private data set SportsAQA and the public data set FR-FS, wherein:

[0110] The data in the private data set SportsAQA is collected from two types of actions in college physical examination, namely pull-ups and sit-ups, wherein there are 4274 pull-up data and 4888 sit-up data, and the training, verification and test sets are randomly selected from the data set according to the ratio of 6:2:2. The annotation uses a three-person cross-validation annotation strategy.

[0111] The public dataset FR-FS is mainly used to evaluate the fall identification of figure skating, containing 417 videos. FR-FS contains the key actions of the athlete's jump, rotation and landing. Among them, there are 276 videos of smooth landing and 141 videos of falling. 50% of the videos are randomly selected from the falling and smooth landing videos as the training set and the test set.

[0112] II. Simulation content

[0113] Simulation 1: On the pull-up data of the private dataset SportsAQA, the video action quality is evaluated using the present application and four common action recognition models respectively, using the F1 score under three different rules R1, R2, R3, and the average F1 score F1_avg of all rules as evaluation indicators, and the results are shown in Table 1.

[0114] Table 1 Evaluation results of pull-up data set

[0115]

[0116] The four action recognition models in Table 1 are Video Swin Transformer, I3D, TimeSformer and R(2+1)D, the F1 score is the harmonic mean of precision and recall, the precision is the proportion of actual positive samples in all predicted positive samples, and the recall is the proportion of correctly predicted positive samples in all actual positive samples. The score is used to measure the balance between accuracy and recall, and can reflect the overall accuracy of the evaluation results, and its formula is:

[0117]

[0118] From Table 1, it can be seen that the F1 score of the method of the present application under R1 and R2 evaluation rules is higher than that of other methods, and the average F1 score reaches the highest 0.9296, showing the best performance on the pull-up data set. In real life, the pull-up rule R2 "the body is assisted by the wave or swing" is a relatively difficult action to judge, and the F1 score of rule R2 of the model of the present application is greatly improved after the guidance of the text rule, with a maximum increase of 3.4%, indicating the effectiveness of the text rule guidance of the present application.

[0119] Simulation 2: On the sit-up data of the private dataset SportsAQA, the video action quality is evaluated using the present application and the four action recognition models used in simulation 1 respectively, using the F1 score under three different rules R1, R2, R3, and the average F1 score F1_avg of all rules as evaluation indicators, and the results are shown in Table 2.

[0120] Table 2 Evaluation results of sit-up data set

[0121]

[0122] As can be seen from Table 2, compared with the Video Swin Transformer model, the average F1 score is improved by 2.25%, thereby verifying the effectiveness of the present application. Although the F1 scores of the conventional action recognition model in rules R1 and R3 can achieve good effects, the conventional action recognition model cannot well predict rule R2, because the ratio of positive and negative samples of rule R2 is 4561 / 324, reaching nearly 14 times, and the multi-modal rule space perception fusion module of the present application well solves this problem, so that the performance of the network on rule 2 is maximally improved by 11.16%.

[0123] Simulation 3: Using the present application and the existing TSA-Net and Plain-Net methods respectively to evaluate the video action quality on the public data set FR-FS, the simulation uses the accuracy as the evaluation index, that is, the ratio of all correct predictions to the total number of all predictions, and the results are shown in Table 3.

[0124] Table 3 Estimation results on the FR-FS data set

[0125]

[0126] As can be seen from Table 3, the Plain-Net method using the conventional I3D feature extraction and direct classification has achieved an accuracy of 94.23%. After introducing the self-attention mechanism based on detailed analysis of the human action region, the accuracy of the TSA-Net method is increased to 98.56%, with an increase of 4.33%. And the present application further increases the accuracy to 99.51% by integrating the time course clue mining and the multi-modal rule space perception fusion module, achieving the best effect on this data set. The performance improvement not only shows the strong ability of the present application in deeply understanding the details of figure skating actions and accurately determining the falling events, but also proves the key role of time course analysis and multi-modal information fusion in improving the accuracy of action evaluation.

[0127] The above simulation results show that the method of the action space-time perception quality evaluation network based on text rule prompt learning of the present application can effectively improve the accuracy and precision of sports fitness action quality evaluation.

[0128] It should be noted that the step labels in the present application specification and claims are only for the clear description of the embodiments of the present application, and are not limited in sequence.

Claims

1. A method for action spatio-temporal perception quality assessment based on text rule prompt learning, characterized in that, The method comprises the following steps: (1) collecting action videos of different fitness methods, labeling according to requirements, and dividing training set and test set according to different proportions according to the category of action; (2) based on the existing CLIP model, a motion spatiotemporal perception quality evaluation network based on text rule prompt learning is constructed: (2a) a fine-grained video spatiotemporal encoder module composed of a video encoder module and a 3D transpose convolution module is established, which is used to extract fine-grained basic video features of the video; The implementation is as follows: (2a1) extracting input video by a video encoder module video features with dimension wherein , is the number of video frames, and denote the height and width of the video, respectively, is the number of channels; (2a2) expanding the temporal channel of the video feature , i.e. upsampling it from to , followed by a 3D transpose convolution module and a global 3D max-pooling operation to obtain a fine-grained base video feature with dimension , where is the number of gym exercise rules; (2b) Establish a time progression cue mining module based on the existing CLIP image encoder and CLIP text encoder, for generating time attention using the correlation between video frames and action progression cue text ; is implemented as follows: (2b1) input the video into the CLIP image encoder to generate a CLIP image visual feature with a dimension of , and simultaneously according to a set of rule texts of the fitness action, construct a text fine-grained description containing all rule text subsets , and input it into the CLIP text encoder to generate a CLIP text feature with a dimension of ; (2b2) splicing the CLIP text features along the time dimension with the CLIP image visual features to generate spliced features with a dimension of Adaptive fusion of the spliced features through the time sequence Transformer to generate CLIP time sequence image features with a dimension of ;​​ (2b3) generating the clue text using the time period information corresponding to the action and the rule and inputting it into the CLIP text encoder to generate learnable clue features with dimensions wherein is the number of clues;​ (2b4) Measure the similarity between CLIP temporal image features using a similarity computation formula and learnable cue features where the th cue text sentence at the th frame is given by the following similarity computation formula: ； wherein , , superscript denotes transposition; (2b5) After similarity calculation on all video frames of the input video, a similarity value is obtained, and a similarity matrix is formed along the time dimension, obtaining the action time process attention corresponding to each clue text sentence :​ ； wherein represents the relevance of all video frames in the input video to the time course information contained in the i-th clue text sentence; represents the relevance of all video frames in the input video to the time course information contained in the i-th clue text sentence; (2c) a multi-modal rule spatiotemporal perception fusion module based on visual and rule text features is established, which is used to utilize visual and text information to extract multi-modal fusion features which have a direct impact on the evaluation results; (2d) sequentially connecting each module established in steps (2a)-(2c), and connecting the multi-modal fusion features to a plurality of multilayer perceptrons with the same number of rules as the fitness exercise, to constitute an action space-time perception quality assessment network based on text rule prompt learning, and taking an existing BCEWithLogitsLoss loss function as the loss function of the network ; (3) input the training set data into the above motion spatiotemporal perception quality evaluation network, and train it by using the AdamW optimization algorithm until the network loss function is in a convergent state, so as to obtain the trained quality evaluation network: (4) input the test set data into the trained motion spatiotemporal perception quality evaluation network to obtain the estimation results corresponding to the test data.

2. The method of claim 1, wherein, Step (1) labels the collected different fitness action videos according to requirements, and divides the training set and test set according to different proportions according to the category of action, which is implemented as follows: (1a) using the Azure Kinect device with a video frame rate of 30 fps and a resolution of 720P, different fitness actions are video collected; (1b) according to the following cross-validation labeling strategy, the collected videos are labeled for each action starting frame, ending frame and action standard degree: First, two volunteers label all video samples respectively, that is, accurately divide and label the starting frame and ending frame of each repeated action in the video sample, and give a standard degree score for each repeated action according to the fitness action rules; Then, the third volunteer cross-verified the labeling results of the first two volunteers, if there is a big difference between the two, the third volunteer needs to re-evaluate the video and modify the labeling; (1c) divide the collected videos into training set and test set, taking pull-up project as an example: For the actions with obvious body displacement and swing amplitude exceeding 30°, and failure rate exceeding 40% in the collected videos, the training set and test set are divided according to the proportion of 4:1; For the actions with small body displacement and swing amplitude not exceeding 30°, and failure rate less than 40% in the collected videos, the training set and test set are divided according to the proportion of 1:

1.

3. The method of any of claims 1-2, wherein, In step (2c), the multi-modal rule spatiotemporal perception fusion module utilizes visual and text information to extract multi-modal fusion features which have a direct impact on the evaluation results, which is implemented as follows: (2c1) multiply the time-course attention generated in step (2b5) with the CLIP time-series image features generated in step (2b2) (2c2) multiply the time-course attention generated in step (2b5) with the CLIP time-series image features generated in step (2b3) (2c3) multiply the time-course attention generated in step (2b5) with the CLIP time-series image features generated in step (2b4) (2c4) multiply the time-course attention generated in step (2b5) with the CLIP time-series image features generated in step (2b5) (2c5) multiply the ； In the formula, wherein is the number of fitness action rules, is the number of input video frames, is the number of channels; represents the CLIP video feature of the rule and the related action process, , The rule corresponds to related video features ; represents whether the action rule contains corresponding time process attention: if it is 1, it indicates that the action rule has corresponding time process attention; otherwise, it has no corresponding time process attention. (2c2) The fine-grained basic video features generated by the fine-grained video encoder module in step (2a2) are processed. With the time-process attention generated in step (2b5) Multiply to obtain CLIP video features Fine-grained video features with time alignment : ； In the formula, , indicates the fine-grained video features corresponding to the rules. (2c3) generating evaluation rule hints of the fitness action around the key evaluation points of the action, and inputting them to the CLIP text encoder to generate learnable rule features with a dimension of , and then copying and extending them to a dimension matched with the CLIP video features , to obtain text features with a dimension of ;​​ (2c4) concatenating the CLIP video features with the text features in the feature dimension, and generating a multimodal query with dimension using a linear projection with weights ; (2c5) performing multi-modal fusion on the multi-modal query with fine-grained video features to obtain fusion features : First, temporal aggregation operations are performed on and to generate more representative query features and fine-grained video features : ； ； Again And After adding by After nonlinear transformation by the activation function, the fusion features with the dimension of :​ ； (2c6) on fusion features The attention scores are calculated by a linear transformation layer and normalized by a Softmax function to get attention weights of dimension :​ ； wherein is a linear projection with weights is a linear projection with weights (2c7) using attention weights Further extract features, extract multi-modal fusion features that have a direct impact on the evaluation results : First, assign attention weights Copying in the time dimension The dimension is obtained as Attention weight matrix Then, they are compared with multimodal query features respectively. and fine-grained video features Multiply to get and : ； ； Again And After splicing in the feature channel, the final multi-modal fusion feature with dimension is generated by linear projection : ； wherein is a linear projection with weights of 1. (2c8) introducing a graph convolution network to the modality fusion feature Extract information, mine the relationship between each rule, and obtain the relationship vector corresponding to the fitness action rule : The multi-modal fusion feature is aggregated in a time dimension to obtain an aggregated fusion feature with a dimension of The multi-modal fusion feature is aggregated in a time dimension to obtain an aggregated fusion feature with a dimension of The multi-modal fusion feature is aggregated in a time dimension to obtain an aggregated fusion feature with a dimension of , and then global representation is obtained after global average pooling and a convolution layer ; global representation extended posterior concatenation of aggregated features intermediate features linear projection with weights and sigmoid activation to obtain a dynamic dependency matrix reflecting the dependency of specific rules : , is a Sigmoid activation function; Aggregating fusion features Input dynamic GCN network, set a static correlation matrix , use the matrix to capture coarse-grained category dependency as the basis of global rule relationship, superimpose it with dynamic correlation matrix After superimposition, output the relationship matrix with dimension of state update weight matrix and LeakyReLU activation function : ； wherein is a LeakyReLU activation function, is a state update weight matrix; The relationship matrix is obtained again After the multilayer perceptron, the relationship vector set corresponding to each piece of fitness action rule is obtained through the Sigmoid activation function : ； In the formula, is the number of fitness action rules, and the dimension of each relationship vector is .

4. The method of claim 1, wherein, In step (2d), the multi-modal fusion features are connected to the multi-layer perceptron with the same number of fitness rules, which is implemented as follows: multimodal fusion features as input to a multilayer perceptron, in accordance with the multimodal fusion features are input to a respective multilayer perceptron, each multilayer perceptron corresponding to evaluation of a respective rule; Fusing features of multiple modalities The output result obtained after each multi-layer perceptron is multiplied by the rule relationship vector corresponding to the multi-layer perceptron to obtain the classification result of each rule.

5. The method of claim 1, wherein, The loss function set for the action-spatiotemporal perception quality assessment network in step (2d) is set as follows in combination with a Sigmoid activation function and a binary cross-entropy loss: ； wherein, is a true value, is a predicted result.

6. The method of claim 1, wherein, In step (3), the training set data is input into the motion spatiotemporal perception quality evaluation network, and the AdamW optimization algorithm is used for training, which is implemented as follows: (3a) Given a video, the fine-grained video encoder module input to the network generates fine-grained basic video features ; (3b) inputting the video simultaneously into a temporal progression cue mining module in the network to generate CLIP temporal image features and temporal progression cue features , performing similarity computation between the two to generate temporal progression attention ; (3c) multiplying the time-course attention with the CLIP temporal image features and the fine-grained base video features to generate CLIP video features and fine-grained video features temporally aligned therewith ; (3d) integrate action rule text description, CLIP video feature and fine-grained video features , generate multi-modal fusion features directly related to the scoring rules through multi-modal rule space perception fusion modules in the network , and perform rule relationship mining to generate relationship vectors ; (3e) the multi-modal fused features and relationship vectors a classification result is computed by a multi-layer perceptron in the network; (3f) using a loss function computing an error of the classification result from the true value, back-propagating the error information, and minimizing adjusting the weight parameters of the network; (3g) repeat steps (3d)~(3f) until the training times reach the set threshold or the value of the loss function converges, and the trained motion spatiotemporal perception quality evaluation network is obtained.

Images