Action space-time perception quality evaluation method based on text rule prompt learning

Abstract

The invention discloses an action space-time perception quality assessment method based on text rule prompt learning, and mainly solves the problems of large assessment limitation and low accuracy in the prior art. According to the implementation scheme, action videos of different fitness methods are collected and marked according to requirements, and a training set and a test set are divided; based on an existing CLIP model, constructing an action space-time perception quality evaluation network based on text rule prompt learning; mining time process clues of action rules, and extracting video action features and rule text features which have direct influence on an evaluation result; performing multi-modal fusion on the video action features and the rule text features; training an action space-time perception quality evaluation network based on text rule prompt learning by using the training set; and inputting test data into the trained quality evaluation network to give an action score, and giving a result whether the action is standard or not. According to the method, the cost of action quality evaluation is reduced, the evaluation accuracy is improved, and the method can be used for fine-grained physical fitness action quality evaluation.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep learning and video understanding, and in particular relates to a video action quality assessment method, which can be used for fine-grained sports fitness action quality assessment. Background Art

[0002] Action quality assessment is a study on the quantification of human performance when performing actions, involving multiple fields such as sports, medicine, and teaching. In sports, the performance assessment of athletes usually relies on experienced coaches or experts in the field. However, this method is limited by the time of experts and the subjectivity of the assessment, and it is difficult to provide immediate, accurate, and consistent feedback. In the medical field, when evaluating the patient's rehabilitation process, the accuracy of surgical operations, and the effect of physical therapy, it is necessary to accurately measure the quality of movement. In traditional medical practice, doctors or therapists usually rely on their own expertise and experience. This method is inefficient, time-consuming, labor-intensive, and susceptible to subjective factors. Therefore, it is necessary to establish a computer vision model to automatically evaluate the quality of human movements in order to accurately capture and analyze subtle differences in movements and provide effective feedback and guidance to humans.

[0003] Movement quality assessment has gradually received widespread attention in recent years, especially in competitive sports such as diving, gymnastics, figure skating and other projects with high precision and expressiveness requirements. Slight differences in movements in these projects may cause significant changes in scores, so it is necessary to accurately assess the quality of athletes' movements. Current research uses rich, easily accessible and authoritative datasets in competitive sports, and mainly focuses on using deep learning regression tasks to predict movement scores. In contrast, in the field of fitness, especially in daily environments, there are fewer related studies. Due to the lack of clear scoring criteria, it is currently challenging to quantify the quality of fitness movements. In daily fitness movement assessment, it is usually necessary to judge whether the movement is standardized based on rules, and this task is essentially a multi-label binary classification task. However, in the field of movement quality assessment, especially in the fine classification of fitness movements, current research is still insufficient.

[0004] The patent document with publication number CN113663312A discloses a method for evaluating the quality of fitness movements without equipment based on micro-inertia. This method adopts a two-step classification of state and action. It first uses three 6-axis micro-inertial sensors to identify the motion state of the human body, then calls the Elman Kalman trajectory estimation model to predict the motion trajectory of each sensor node, and then uses the trajectory signal and the collected signal for the second step of fitness action classification. After the classification is completed, the generated motion sequence is compared and analyzed with the sequences in the standard motion sequence library to evaluate the whole-body and local standardization and stability of the action, and give the action quality evaluation results. However, due to the use of 6-axis micro-inertial sensors, this method has a high cost and low evaluation accuracy. At the same time, it limits the identifiable action categories and cannot be used in all scenarios.

[0005] The patent document with the publication number CN113255489A discloses a multimodal intelligent evaluation method for diving events based on labeled distribution learning. This method is based on a bimodal inflated 3D ConvNet (I3D), adds an optical flow, extracts the spatiotemporal features of the diving video in RGB and Flow modes, and fuses them into a global feature to input it into the fully connected layer. The labeled distribution learning LDL is used to output the probability distribution of the mean of all referee scores, and then the maximum probability score is selected and multiplied by the difficulty coefficient to obtain the final diving score. However, this method can only realize the quality evaluation of diving, and because it uses optical flow extraction, not only is the network implementation process complicated, but the running speed is also slow.

[0006] In addition, the above existing methods only focus on the visual information during fitness exercises, do not analyze the spatial features related to the evaluation rules, and ignore the role of the time process clues of the action rule text at the key moments of the exercise. As a result, subtle action differences cannot be found during the exercise, resulting in poor evaluation accuracy. Summary of the invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a method for evaluating the spatiotemporal perception quality of actions based on text rule prompt learning. According to the time process clues of the action rule text and the time relationship between the action process and the score, the key moments that have a decisive influence on the action score are identified, and these key moments are used to improve the accuracy of action evaluation.

[0008] To achieve the above object, the implementation scheme of the present invention includes the following steps:

[0009] (1) Collect action videos of different fitness methods, annotate them according to requirements, and divide them into training sets and test sets according to different proportions according to the types of actions;

[0010] (2) Based on the existing CLIP model, a spatiotemporal action perception quality assessment network based on text rule prompt learning is constructed:

[0011] (2a) establishing a fine-grained video spatiotemporal encoder module consisting of a video encoder module and a 3D transposed convolution module to extract fine-grained basic video features of the video;

[0012] (2b) Establish a temporal process clue mining module based on the existing CLIP image encoder and CLIP text encoder to generate temporal attention Simulation using the correlation between video frames and action process clue text n ;

[0013] (2c) Establish a multimodal rule-space perception fusion module based on visual and rule-text features to use visual and text information to extract multimodal fusion features that have a direct impact on the evaluation results;

[0014] (2d) Cascade the modules established in steps (2a) to (2c) in sequence, and connect the multimodal fusion features to multilayer perceptrons with the same number as the fitness exercise rules to form a spatiotemporal perception quality assessment network based on text rule prompt learning, and use the existing BCEWithLogitsLoss loss function as the loss function Loss of the network;

[0015] (3) Input the training set data into the above-mentioned action spatiotemporal perception quality assessment network and train it using the AdamW optimization algorithm until the network loss function is in a convergence state, thereby obtaining a trained quality assessment network:

[0016] (4) Input the test set data into the trained motion spatiotemporal perception quality assessment network to obtain the estimation results corresponding to the test data.

[0017] Compared with the prior art, the present invention has the following advantages:

[0018] First, increase the network's ability to learn action details.

[0019] Since the present invention constructs a time process clue mining module, it can help the network better understand the subtle differences in action details during fitness exercises by mining the time process clue information of action rules, thereby improving the accuracy of evaluation. At the same time, since the present invention constructs a multimodal rule space perception fusion module, it can further mine visual and rule text features, extract action detail features that have a direct impact on the evaluation results, and make the action detail features easier to be learned by the network.

[0020] Second, improve the accuracy and precision of action quality assessment.

[0021] The present invention obtains fine-grained video features by introducing a corresponding number of 3D transposed convolutions and global 3D maximum pooling operations for each rule of fitness movements, and combines textual spatiotemporal prompts to conduct deeper spatiotemporal information mining on the obtained video features. At the same time, a graph convolutional network is introduced to alleviate the problems caused by label imbalance, thereby improving the accuracy and precision of the evaluation quality.

[0022] Third, reduce the cost of use.

[0023] The present invention performs spatiotemporal quality assessment of actions based on common RGB videos, so the hardware equipment required is low-cost and suitable for large-scale promotion and application. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the implementation of the present invention;

[0025] Figure 2 is a network structure diagram of action spatiotemporal perception quality assessment in the present invention;

[0026] Figure 3 yes Figure 2 Diagram of the rule relationship mining structure. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, other embodiments obtained by ordinary technicians in this field without creative work should all fall within the scope of protection of the present invention.

[0028] Reference Figure 1 , the implementation steps of this example are as follows:

[0029] Step 1. Label the collected videos of different fitness movements as required, and divide the training set and test set into different proportions according to the movement categories.

[0030] (1.1) Use the Azure Kinect device with a video frame rate of 30fps and a resolution of 720P to capture videos of different fitness movements;

[0031] (1.2) Annotate the start frame, end frame, and standard degree of each action in the captured video according to the following cross-validation annotation strategy:

[0032] (1.2.1) Two volunteers will annotate all video samples, i.e. accurately mark the start and end frames of each repetitive action in the video sample, and give a standard score for each repetitive action according to the fitness action rules;

[0033] (1.2.2) A third volunteer cross-validates the annotation results of the first two volunteers. If a significant difference is found between the two, the third volunteer will re-evaluate the video and modify its annotations;

[0034] (1.3) Divide the collected videos into training sets and test sets, taking the pull-up project as an example:

[0035] For actions with obvious body displacement, swing amplitude exceeding 30°, and error rate exceeding 40% in the collected videos, the training set and test set are divided into training set and test set in a ratio of 4:1;

[0036] For actions in the collected videos with small body displacement, swing amplitude not exceeding 30°, and error rate below 40%, the training set and test set are divided into training set and test set in a 1:1 ratio.

[0037] Step 2. Construct an action spatiotemporal perception quality assessment network based on text rule prompt learning.

[0038] Reference Figure 2 , the specific implementation of this step is as follows:

[0039] (2.1) Use the fine-grained video spatiotemporal encoder to extract the fine-grained basic video features of the video:

[0040] (2.1.1) Extract the video features F of the input video V through the video encoder module v , whose dimensions are where V∈R T×H×W×3 , T is the number of video frames, H and W represent the height and width of the video respectively, and D is the number of channels;

[0041] (2.1.2) Video feature F v The temporal channel of is expanded, that is, it is upsampled from T / 2 to T, and then the fine-grained basic video feature F is obtained through the 3D transposed convolution module and the global 3D maximum pooling operation. rule , whose dimension is M×T×D, where M is the number of fitness action rules;

[0042] (2.2) Use the temporal process clue mining module to generate temporal attention Sim based on the correlation between video frames and action process clue text n :

[0043] (2.2.1) Input the video V into the CLIP image encoder to generate a CLIP image visual feature F with a dimension of T×D cv At the same time, according to the M fitness action rule texts, a text fine-grained description T containing all rule text subsets is constructed d , and input it into the CLIP text encoder to generate a dimension of 2 M ×D CLIP text feature F ct ;

[0044] (2.2.2) The CLIP text feature F is transformed along the time dimension ct and CLIP image visual features F cv Splice and generate a dimension of (T+2 M )×D splicing feature F c The concatenated features are adaptively fused through the temporal Transformer to generate a CLIP temporal image feature F with a dimension of T×D image ;

[0045] (2.2.3) Generate clue text T using the time period information corresponding to the action and rule n , and input it into the CLIP text encoder to generate a learnable clue feature F with dimension N×D text , where N is the number of clues;

[0046] (2.2.4) Using the similarity calculation formula to measure the CLIP time series image feature F image and learnable clue features F text The similarity between the nth clue text sentence in the tth frame is calculated as follows:

[0047]

[0048] Where, t∈[1,T], n∈[1,N], and the superscript ' indicates transposition;

[0049] (2.2.5) Calculate the similarity of all video frames of the input video to obtain T similarity values, and combine them into a similarity matrix along the time dimension to obtain the action time process attention Sim corresponding to each clue text sentence n :

[0050]

[0051] In the formula, Sim n ∈R T , which represents the correlation between all video frames in the input video and the time process information contained in the nth clue text sentence;

[0052] (2.3) Using the multimodal rule-based spatial perception fusion module, visual and textual information is used to extract multimodal fusion features that have a direct impact on the evaluation results:

[0053] (2.3.1) The time process attention Sim generated in step (2.2.5) n The CLIP time series image feature F generated in step (2.2.2) image Multiply them to get the CLIP video feature F which is directly related to the action process clip :

[0054]

[0055] In the formula, F clip ∈R M×T×D , M is the number of fitness action rules, T is the number of input video frames, and D is the number of channels; F clip m =F image ·(1-Y m,n )+F image ·Y m,n ·Sim n Represents the CLIP video features of the mth rule and the related action process, m∈{1,2,...,M}, M rules correspond to M related video features F clip (1)～(M) ; Y m,n ∈{0,1} indicates whether the action rule m contains the corresponding time process attention: if it is 1, it means that the action rule has the corresponding time process attention; otherwise, there is no corresponding time process attention;

[0056] (2.3.2) The fine-grained basic video features F generated by the fine-grained video encoder module in step (2.1.2) are rule The time course attention Sim generated in step (2.2.5) n Multiply it by the CLIP video feature F clip Temporally aligned fine-grained video features F align :

[0057] F align ={F align (1) ,F align (2) ,...,F align (m) ,...,F align (M)}

[0058] In the formula, F align ∈R M×T×D , F alignm =F rule m ·(1-Y m,n )+F rule m ·Y m,n ·Sim n represents the fine-grained video features corresponding to the mth rule;

[0059] (2.3.3) Generate evaluation rule prompts for fitness actions around the key evaluation points of the action and input them into the CLIP text encoder to generate a learnable rule feature F with dimension M×D eva , and then replicate and extend it to the CLIP video feature F clip The matching dimensions give a text feature F with dimensions M×T×D eva ′;

[0060] (2.3.4) The CLIP video feature F clip With the text feature F eva ′ is concatenated in the feature D dimension and uses the weight W q ∈R 2D×D The linear projection of the generated multimodal query Q = Concat(F clip ,F eva ′)W q ;

[0061] (2.3.5) Combine the multimodal query Q with the fine-grained video feature F align Perform multimodal fusion to obtain the fusion feature H:

[0062] First, for Q and F align Perform temporal aggregation operations to generate more representative query features Q′ and fine-grained video features F align ′:

[0063] Q′=AggregateT(Q)

[0064] F align ′=AggregateT(F align )

[0065] Then Q′ and F align After adding them, they are transformed nonlinearly through the tanh activation function to obtain the fusion feature H with a dimension of M×D:

[0066] H = tanh(Q′ + F align ′);

[0067] (2.3.6) The attention score of the fused feature H is calculated through the linear transformation layer and normalized through the Softmax function to obtain the attention weight A with dimension M×D:

[0068] A=Softmax(HW a )

[0069] Where W a The weight is R D×D Linear projection of

[0070] (2.3.7) Use the attention weight A to further extract features and extract the multimodal fusion feature F that has a direct impact on the evaluation results:

[0071] First, the attention weight A is copied to T dimension in the time dimension to obtain the attention weight matrix A′ with the dimension of M×T×D, and then it is combined with the multimodal query feature Q and the fine-grained video feature F respectively. align Multiply them to get Q″ and F align ″:

[0072] Q″=A′⊙Q

[0073] F align ″=A′⊙F align

[0074] Then Q″ and F align After concatenating the feature channels, a linear projection is performed to generate the final multimodal fusion feature F with a dimension of M×T×D:

[0075] F=Concat(Q″,F align ″)W f

[0076] Where W f The weight is R 2D×D Linear projection of

[0077] (2.3.8) Introduce graph convolutional network to extract information from modal fusion feature F, explore the relationship between each rule, and obtain the relationship vector Vector corresponding to the fitness action rule 1-M :

[0078] Reference Figure 3 , the specific implementation of this step is as follows:

[0079] The multimodal fusion feature F is aggregated in the time dimension to obtain an aggregated fusion feature F′ with a dimension of M×D, and then it is passed through a global average pooling and a convolution layer to obtain the global representation F g ∈R D ;

[0080] The global representation Fg After expanding M dimensions, the fused features F′ are aggregated in series to obtain the intermediate feature F″∈R 2D×M , and then weight it as W A ∈R D×D After linear projection, the dynamic correlation matrix reflecting the dependency of specific rules is obtained through the Sigmoid activation function: d =δ(W A F″), where δ(·) is the Sigmoid activation function;

[0081] Input the aggregated fusion feature F′ into the dynamic GCN network and set a static correlation matrix A s , using this matrix to capture the coarse-grained category dependency as the basis for global rule relationships, and then combining it with the dynamic correlation matrix A d After superposition, the state update weight matrix and the LeakyReLU activation function output the relationship matrix P with a dimension of M×D:

[0082] P=f((A s +A d )F′W d )

[0083] Where f(·) is the LeakyReLU activation function, W d ∈R D×D Update the weight matrix for the state;

[0084] Then the relationship matrix P is passed through the multi-layer perceptron and the Sigmoid activation function to obtain the relationship vector set Vector corresponding to each fitness action rule 1-M :

[0085] Vector 1-M ={Vector1,Vector2,...,Vector m ,...,Vector M}

[0086] In the formula, M is the number of fitness action rules, and the dimension of each relationship vector is M×1;

[0087] (2.4) Connect the multimodal fusion features to the same number of multilayer perceptrons as the number of fitness rules to obtain the classification results of each rule:

[0088] The multimodal fusion feature F is used as the input of the multilayer perceptron. According to the M rules of the fitness action in the video, the multimodal fusion feature F is input into M multilayer perceptrons respectively, and each multilayer perceptron is responsible for the evaluation of one rule.

[0089] The relationship vector {Vector1, Vector2, ..., Vector M} to obtain the classification result of each rule;

[0090] (2.5) cascading the modules established in steps (2.1) to (2.4) to form a spatiotemporal action perception quality assessment network based on text rule prompt learning;

[0091] (2.6) The existing Sigmoid activation function and binary cross entropy loss are combined to form the loss function Loss of the action spatiotemporal perception quality assessment network:

[0092]

[0093] Among them, y is the true value, For the prediction results.

[0094] Step 3. Input the training set data into the action spatiotemporal perception quality assessment network and train it using the AdamW optimization algorithm.

[0095] (3.1) Given a video, the fine-grained video encoder module input into the network generates fine-grained basic video features F rule ;

[0096] (3.2) The video is simultaneously input into the time process clue mining module of the network to generate CLIP time series image features F image and time course cue feature F text , perform similarity calculation between the two to generate the time process attention Sim;

[0097] (3.3) Combine the time process attention Sim with the CLIP time series image feature F image and fine-grained basic video features F rule Multiply to generate CLIP video feature F clip and the fine-grained video features F that are time-aligned with it align ;

[0098] (3.4) Integrate action rule text description and CLIP video feature F clip and fine-grained video features F align , the multimodal rule space perception fusion module in the network generates a multimodal fusion feature F that is directly related to the scoring rule, and performs rule relationship mining on it to generate a relationship vector Vector 1-M ;

[0099] (3.5) Multimodal fusion feature F and relationship vector Vector1-M The classification result is calculated through the multi-layer perceptron in the network;

[0100] (3.6) Use the loss function Loss to calculate the error between the classification result and the true value, back-propagate the error information, and adjust the weight parameters of the network by minimizing Loss;

[0101] (3.7) Repeat steps (3.4) to (3.6) until the number of training times reaches the set threshold or the value of the loss function converges, and a trained motion spatiotemporal perception quality assessment network is obtained.

[0102] Step 4. Input the test data set into the trained action spatiotemporal perception quality assessment network to obtain the evaluation results corresponding to the test data:

[0103] (4.1) Randomly select b video samples from the test set data in batches, and pre-set M action rule texts for each video according to its video action category. Input them and the video samples into the network together, and output M action rule quality evaluation results corresponding to the M action rule texts. Each evaluation result can only be standard or non-standard;

[0104] (4.2) Output the final action evaluation quality of the test video based on the action rule quality evaluation result:

[0105] When the quality evaluation results of M action rules are all standard, the quality evaluation result of the video action is output as standard;

[0106] On the contrary, if there is a non-standard result among the M action rule quality evaluation results, then the quality evaluation result of the video action is judged to be in violation of the rules.

[0107] The effect of the present invention can be further illustrated by the following simulation.

[0108] 1. Simulation Data

[0109] Two datasets are used: the private dataset SportsAQA and the public dataset FR-FS, where:

[0110] The data in the private dataset SportsAQA are collected from two types of movements in college physical tests, namely pull-ups and sit-ups. There are 4274 pull-up data and 4888 sit-up data. The training, validation and test sets are randomly selected from the dataset in a ratio of 6:2:2. The annotation uses a three-person cross-validation annotation strategy.

[0111] The public dataset FR-FS is mainly used to evaluate figure skating fall recognition and contains 417 videos. FR-FS contains the key actions of athletes taking off, rotating and landing. There are 276 videos of smooth landing and 141 videos of falls. 50% of the videos are randomly selected from the fall and smooth landing videos as training and test sets.

[0112] 2. Simulation Content

[0113] Simulation 1, on the pull-up data of the private dataset SportsAQA, the present invention and four common action recognition models are used to evaluate the quality of video action, and the F1 scores under three different rules R1, R2, and R3, as well as the average F1 score F1_avg of all rules are used as evaluation indicators. The results are shown in Table 1.

[0114] Table 1 Evaluation results of pull-up dataset

[0115]

[0116] The four action recognition models in Table 1 are Video Swin Transformer, I3D, TimeSformer and R(2+1)D. The F1 score is the harmonic mean of precision and recall. Precision is the proportion of samples predicted as positive that are actually positive, and recall is the proportion of samples actually positive that are correctly predicted as positive. This score is used to measure the balance between the accuracy and recall of the model, which can reflect the overall accuracy of the evaluation results. The formula is:

[0117]

[0118] As can be seen from Table 1, the F1 scores of the method of the present invention under the R1 and R2 evaluation rules are higher than those of other methods, and the average F1 score reaches the highest 0.9296, showing the best performance on the pull-up dataset. In real life, the pull-up rule R2 "body with the help of waves or swings" is a relatively difficult action to judge. After adding the guidance of text rules, the F1 score of rule R2 of the model of the present invention has been greatly improved, with a maximum increase of 3.4%, indicating the effectiveness of the text rule guidance of the present invention.

[0119] Simulation 2, on the sit-up data of the private dataset SportsAQA, the four action recognition models used in the present invention and simulation 1 are used to evaluate the video action quality, and the F1 scores under three different rules R1, R2, and R3, as well as the average F1 score F1_avg of all rules are used as evaluation indicators. The results are shown in Table 2;

[0120] Table 2 Evaluation results of sit-up dataset

[0121]

[0122] As can be seen from Table 2, compared with the Video Swin Transformer model, the present invention achieves a 2.25% improvement in the average F1 score, thereby verifying the effectiveness of the present invention. Although the conventional action recognition model can achieve relatively good results in the F1 scores of rules R1 and R3, the conventional action recognition model cannot make a good prediction for rule R2. This is because the ratio of positive and negative samples in rule R2 is 4561 / 324, which is nearly 14 times. The multimodal rule space perception fusion module of the present invention solves this problem well, so that the network performance on rule 2 is improved by 11.16% at most.

[0123] Simulation 3, on the public dataset FR-FS, the video motion quality is evaluated using the present invention and the existing TSA-Net and Plain-Net methods respectively. The simulation uses accuracy, that is, the proportion of all correctly classified predictions to the total number of all predictions, as the evaluation indicator. The results are shown in Table 3.

[0124] Table 3 Estimation results on the FR-FS dataset

[0125]

[0126] As can be seen from Table 3, the Plain-Net method that uses traditional I3D feature extraction and direct classification has achieved an accuracy of 94.23%. After introducing a self-attention mechanism to conduct a detailed analysis of the human motion area, the accuracy of the TSA-Net method increased to 98.56%, an increase of 4.33%. The present invention further increased the accuracy to 99.51% by integrating time process clue mining and multimodal rule space perception fusion modules, achieving the best results on this dataset. This performance improvement not only demonstrates the powerful ability of the present invention in deeply understanding the details of figure skating movements and accurately determining fall events, but also proves the key role of time process analysis and multimodal information fusion in improving the accuracy of movement evaluation.

[0127] The above simulation results show that the method of the present invention can effectively improve the accuracy and precision of sports fitness action quality assessment through the action spatiotemporal perception quality assessment network method based on text rule prompt learning.

[0128] It should be noted that the step numbers in the specification and claims of the present invention and in the claims are only for a clear description of the implementation scheme of the present invention and for ease of understanding, and the sequence of the numbers is not limited.

Claims

1. A method for evaluating the spatiotemporal perception quality of actions based on text rule prompt learning, characterized in that: include: (1) Collect action videos of different fitness methods, annotate them according to requirements, and divide them into training sets and test sets according to different proportions according to the types of actions; (2) Based on the existing CLIP model, a spatiotemporal action perception quality assessment network based on text rule prompt learning is constructed: (2a) establishing a fine-grained video spatiotemporal encoder module consisting of a video encoder module and a 3D transposed convolution module to extract fine-grained basic video features of the video; (2b) Establish a temporal process clue mining module based on the existing CLIP image encoder and CLIP text encoder to generate temporal attention Simulation using the correlation between video frames and action process clue text n ; (2c) Establish a multimodal rule-space perception fusion module based on visual and rule-text features to use visual and text information to extract multimodal fusion features that have a direct impact on the evaluation results; (2d) Cascade the modules established in steps (2a) to (2c) in sequence, and connect the multimodal fusion features to multilayer perceptrons with the same number as the fitness exercise rules to form a spatiotemporal perception quality assessment network based on text rule prompt learning, and use the existing BCEWithLogitsLoss loss function as the loss function Loss of the network; (3) Input the training set data into the above-mentioned action spatiotemporal perception quality assessment network and train it using the AdamW optimization algorithm until the network loss function is in a convergence state, thereby obtaining a trained quality assessment network: (4) Input the test set data into the trained motion spatiotemporal perception quality assessment network to obtain the estimation results corresponding to the test data.

2. The method according to claim 1, characterized in that Step (1) annotates the collected videos of different fitness movements as required, and divides the training set and the test set into different proportions according to the types of movements, as follows: (1a) Use the Azure Kinect device with a video frame rate of 30fps and a resolution of 720P to capture videos of different fitness movements; (1b) Annotate the start frame, end frame, and standard degree of each action in the captured video according to the following cross-validation annotation strategy: First, two volunteers labeled all the video samples separately, that is, accurately divided and labeled the starting frame and ending frame of each repeated action in the video sample, and gave a standard score for each repeated action under each rule according to the fitness action rules. Then, the third volunteer cross-validates the annotation results of the first two volunteers. If a large difference is found between the two annotation results, the third volunteer needs to re-evaluate the video and modify its annotation; (1c) Divide the collected videos into training sets and test sets, taking the pull-up project as an example: For actions with obvious body displacement, swing amplitude exceeding 30°, and error rate exceeding 40% in the collected videos, the training set and test set are divided into training set and test set in a ratio of 4:1; For actions in the collected videos with small body displacement, swing amplitude not exceeding 30°, and error rate below 40%, the training set and test set are divided into training set and test set in a 1:1 ratio.

3. The method according to claim 1, characterized in that The fine-grained video spatiotemporal encoder described in step (2a) is used to extract fine-grained basic video features of the video, and its implementation is as follows: (2a1) Extract the video features F of the input video V through the video encoder module v , whose dimensions are where V∈R T×H×W×3 , T is the number of video frames, H and W represent the height and width of the video respectively, and D is the number of channels; (2a2) For video feature F v The temporal channel of is expanded, that is, it is upsampled from T / 2 to T, and then the fine-grained basic video feature F is obtained through the 3D transposed convolution module and the global 3D maximum pooling operation. rule , whose dimension is M×T×D, where M is the number of fitness action rules.

4. The method according to claim 1, characterized in that: The temporal process clue mining module described in step (2b) generates a temporal attention Sim using the correlation between the video frame and the action process clue text n , implemented as follows: (2b1) Input the video V into the CLIP image encoder to generate a CLIP image visual feature F with a dimension of T×D cv , and at the same time, based on the M fitness action rule texts, a text fine-grained description T containing all rule text subsets is constructed d , and input it into the CLIP text encoder to generate a dimension of 2 M ×D CLIP text feature F ct ; (2b2) ​​Transform the CLIP text feature F along the time dimension ct and CLIP image visual features F cv Splice and generate a dimension of (T+2 M )×D splicing feature F c The concatenated features are adaptively fused through the temporal Transformer to generate a CLIP temporal image feature F with a dimension of T×D image ; (2b3) Generate clue text T using the time period information corresponding to the action and the rule n , and input it into the CLIP text encoder to generate a learnable clue feature F with dimension N×D text , where N is the number of clues; (2b4) Use the similarity calculation formula to measure the CLIP time series image feature F image and learnable clue features F text The similarity between the nth clue text sentence in the tth frame is calculated as follows: Where, t∈[1,T], n∈[1,N], and the superscript ' indicates transposition; (2b5) After similarity calculation for all video frames of the input video, T similarity values ​​are obtained, and they are combined into a similarity matrix along the time dimension to obtain the action time process attention Sim corresponding to each clue text sentence n : In the formula, Sim n ∈R T , represents the correlation between all video frames in the input video and the time process information contained in the nth clue text sentence.

5. The method according to any one of claims 1 to 4, characterized in that: The multimodal rule-space perception fusion module described in step (2c) uses visual and textual information to extract multimodal fusion features that have a direct impact on the evaluation results, which is implemented as follows: (2c1) The time process attention Sim generated in step (2b5) n and the CLIP time series image feature F generated in step (2b2) image Multiply them to get the CLIP video feature F which is directly related to the action process clip : F clip ={F clip (1) ,F clip (2) ,...,F clip (m) ,...,F clip (M) } In the formula, F clip ∈R M×T×D , where M is the number of fitness action rules, T is the number of input video frames, and D is the number of channels; F clip m =F image ·(1-Y m,n )+F image ·Y m,n ·Sim n Represents the CLIP video features of the mth rule and the related action process, m∈{1,2,...,M}, M rules correspond to M related video features F clip (1)～(M) ; Y m,n ∈{0,1} indicates whether the action rule m contains the corresponding time process attention: if it is 1, it means that the action rule has the corresponding time process attention; otherwise, there is no corresponding time process attention; (2c2) The fine-grained basic video features F generated by the fine-grained video encoder module in step (2a2) are rule The time course attention Sim generated in step (2b5) n Multiply it by the CLIP video feature F clip Temporally aligned fine-grained video features F align : F align ={F align (1) ,F align (2) ,...,F align (m) ,...,F align (M) }; In the formula, F align ∈R M×T×D , F align m =F rule m ·(1-Y m,n )+F rule m ·Y m,n ·Sim n represents the fine-grained video features corresponding to the mth rule; (2c3) Generate evaluation rule prompts for fitness actions around the key evaluation points of the action and input them into the CLIP text encoder to generate a learnable rule feature F with dimension M×D eva , and then replicate and extend it to the CLIP video feature F clip The matching dimensions give a text feature F with dimensions M×T×D eva ′; (2c4) The CLIP video feature F clip With the text feature F eva ′ is concatenated in the feature D dimension and uses the weight W q ∈R 2D×D The linear projection of the generated multimodal query Q = Concat(F clip ,F eva ′)W q ; (2c5) Combine the multimodal query Q with the fine-grained video feature F align Perform multimodal fusion to obtain the fusion feature H: First, for Q and F align Perform temporal aggregation operations to generate more representative query features Q′ and fine-grained video features F align ′: Q′=AggregateT(Q) F align ′=AggregateT(F align ) Then Q′ and F align After adding them together, the tanh activation function is used to perform nonlinear transformation to obtain the fusion feature H with a dimension of M×D: H=tanh(Q′+F align ′); (2c6) The attention score of the fused feature H is calculated through a linear transformation layer and normalized through the Softmax function to obtain the attention weight A with a dimension of M×D: A=Softmax(HW a ) Where W a The weight is R D×D Linear projection of (2c7) The attention weight A is used to further extract features and extract the multimodal fusion feature F which has a direct impact on the evaluation results: First, the attention weight A is copied to T dimension in the time dimension to obtain the attention weight matrix A′ with the dimension of M×T×D, and then it is combined with the multimodal query feature Q and the fine-grained video feature F respectively. align Multiply them to get Q″ and F align ″: Q″=A′⊙Q F align ″=A′⊙F align Then Q″ and F align After concatenating the feature channels, a linear projection is performed to generate the final multimodal fusion feature F with a dimension of M×T×D: F=Concat(Q″,F align ″)W f Where W f The weight is R 2D×D Linear projection of (2c8) Introduce graph convolutional network to extract information from modal fusion feature F, explore the relationship between each rule, and obtain the relationship vector Vector corresponding to the fitness action rule 1-M : The multimodal fusion feature F is aggregated in the time dimension to obtain an aggregated fusion feature F′ with a dimension of M×D, and then it is passed through a global average pooling and a convolution layer to obtain the global representation F g ∈R D ; The global representation F g After expanding M dimensions, the fused features F′ are aggregated in series to obtain the intermediate feature F″∈R 2D×M , and then weight it as W A ∈R D×D After linear projection, the dynamic correlation matrix A reflecting the dependency of specific rules is obtained through the Sigmoid activation function. d : A d =δ(W A F″), δ(·) is the Sigmoid activation function; Input the aggregated fusion feature F′ into the dynamic GCN network and set a static correlation matrix A s , using this matrix to capture the coarse-grained category dependency as the basis for global rule relationships, and combining it with the dynamic correlation matrix A d After superposition, the state update weight matrix and the LeakyReLU activation function output the relationship matrix P with a dimension of M×D: P=f((A s +A d )F′W d ) Where f(·) is the LeakyReLU activation function, W d ∈R D×D Update the weight matrix for the state; Then the relationship matrix P is passed through the multi-layer perceptron and the Sigmoid activation function to obtain the relationship vector set Vector corresponding to each fitness action rule 1-M : Vector 1-M ={Vector1,Vector2,...,Vector m ,...,Vector M } Where M is the number of fitness action rules, and the dimension of each relationship vector is M×1.

6. The method according to claim 1, characterized in that In step (2d), the multimodal fusion features are connected to the same number of multilayer perceptrons as the number of fitness exercise rules, which can be implemented as follows: The multimodal fusion feature F is used as the input of the multilayer perceptron. According to the M rules of the fitness action in the video, the multimodal fusion feature F is input into M multilayer perceptrons respectively, and each multilayer perceptron is responsible for the evaluation of one rule. The relationship vector {Vector1, Vector2, ..., Vector M } to obtain the classification result of each rule.

7. The method according to claim 1, characterized in that The loss function Loss set for the action spatiotemporal perception quality assessment network in step (2d) is set by combining the Sigmoid activation function and the binary cross entropy loss as follows: Among them, y is the true value, For the prediction results.

8. The method according to claim 1, characterized in that Step (3) inputs the training set data into the action spatiotemporal perception quality assessment network and trains it using the AdamW optimization algorithm, as follows: (3a) Given a video, the fine-grained video encoder module input into the network generates fine-grained basic video features F rule ; (3b) The video is simultaneously input into the time process clue mining module of the network to generate CLIP time series image features F image and time course cue feature F text , perform similarity calculation between the two to generate the time process attention Sim; (3c) Combine the time process attention Sim with the CLIP temporal image feature F image and fine-grained basic video features F rule Multiply to generate CLIP video feature F clip and the fine-grained video features F that are time-aligned with it align ; (3d) Integrate action rule text description and CLIP video feature F clip and fine-grained video features F align , the multimodal rule space perception fusion module in the network generates a multimodal fusion feature F that is directly related to the scoring rule, and performs rule relationship mining on it to generate a relationship vector Vector 1-M ; (3e) The multimodal fusion feature F and the relationship vector Vector 1-M The classification result is calculated through the multi-layer perceptron in the network; (3f) Use the loss function Loss to calculate the error between the classification result and the true value, back-propagate the error information, and adjust the weight parameters of the network by minimizing Loss; (3g) Repeat steps (3d) to (3f) until the number of training times reaches a set threshold or the value of the loss function converges, thereby obtaining a trained motion spatiotemporal perception quality assessment network.

Images