Action quality assessment method and system for cross-modal time sequence semantic alignment

Abstract

The invention provides an action quality evaluation method and system for cross-modal time sequence semantic alignment. The method comprises the following steps: respectively extracting a visual feature Fv and a text feature Ft from a motion sequence video and a motion sequence text description through a regression model; self-supervised sub-action analysis is carried out on the visual feature Fv, and cross-modal attention interaction learning is carried out on the visual feature Fv and the text feature Ft; carrying out cross-modal semantic alignment by adopting a self-supervision mode and carrying out global semantic contrast learning; and performing quality evaluation on the action by adopting a method based on fractional distribution regression. The high-level semantic representation of the sub-action sequence is enhanced by mining the alignment relationship between different modes in the video sequence and sequence internal time sequence association, so that an accurate motion sequence quality evaluation result is obtained; interaction between different modes is promoted through action semantic alignment, and the accuracy of semantic judgment of sub-action sequences is improved; and the accuracy and interpretability of score prediction in motion quality evaluation are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of motion quality assessment, and in particular to a method and system for motion quality assessment based on cross-modal temporal semantic alignment. Background Art

[0002] The Action Quality Assessment (AQA) task aims to design an intelligent scoring system that can automatically and objectively evaluate scores for competitive sports and various movements requiring standardization. Action quality assessment has a wide range of applications, including surgical skill rating and medical rehabilitation training in healthcare, posture correction training systems for athletes in sports, AI refereeing systems in competitions, and operational compliance analysis and risk behavior monitoring in industrial production. The development of artificial intelligence (AI) technology is further promoting the widespread application of action quality assessment technology in a wider range of scenarios.

[0003] Action Quality Assessment (AQA) requires objective scoring of competitive sports and various actions that require standardization. Therefore, AQA also faces some unique challenges, especially when dealing with fine-grained action features (such as "forward", "somersault", "twist" and "entering the water"). Most AQA methods rely on visual features that represent the entire video. On the one hand, foreground action information and background information are treated indiscriminately, ignoring the importance of action features in the video; on the other hand, the lack of analysis of fine-grained sub-actions also greatly weakens the accuracy of action quality assessment. This approach fails to fully utilize action semantic information and limits its generalization ability. This patent proposes a cross-modal temporal semantic interaction method to mine the alignment relationship between different modalities in a video sequence and the internal temporal association of the sequence, enhance the high-level semantic representation of the sub-action sequence, and obtain accurate motion sequence quality assessment results. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a motion quality assessment method and system with cross-modal temporal semantic alignment, which further improves the accuracy and interpretability of score prediction in motion quality assessment.

[0005] The technical solution of the present invention is: a method for evaluating action quality of cross-modal temporal semantic alignment, comprising the following steps:

[0006] S1) Obtain the user's motion sequence video data; extract visual features F from the motion sequence video and the motion sequence text description respectively v and text features F t ;

[0007] S2), visual feature F v Perform self-supervised sub-action parsing to obtain enhanced visual sub-action features;

[0008] S3), visual feature F v and text features F t Conduct cross-modal attention interaction learning;

[0009] S4) Use self-supervision to perform cross-modal semantic alignment and global semantic contrast learning;

[0010] S5) Use the method based on score distribution regression to evaluate the quality of the action.

[0011] As a preferred embodiment, in step S1), the visual feature extraction network P3D is used to extract the motion sequence X. v Extract visual features F v ; Extract text features F from the motion sequence text description through the text feature extraction network BERT t .

[0012] Preferably, in step S2), a self-attention encoder is used to aggregate the temporal feature expression of the action sequence, so that the model can learn the global context-related features of the action sequence to enhance the segment-level feature representation, and the aggregated weights are learned through the correlation between the segments; and a multi-layer Transformer decoder is used as the baseline network for self-attention calculation, and S learnable query The memory vector is composed of the visual features F v and its projection matrix V v Act as,and perform self-attention calculation.

[0013] As a preference, in step S2), for the v The video of the sequence clips is processed by using the learned self-attention matrix to focus on the visual features F v Perform enhancements, namely:

[0014]

[0015] Where, represents the enhanced visual sub-action feature; T v Indicates the number of action segments in the video sequence; represents the self-attention weight matrix; V v is the visual feature F v The projection matrix.

[0016] As a preference, in step S3), during the cross-modal attention interaction learning process, the text feature F t The projection matrix Q t Used as a text context query As the query vector input of the cross attention mechanism, the visual features F vActing as a memory; this integration enables the regression model to optimize the representation of visual features by incorporating relevant text and its contextual key descriptions.

[0017] Preferably, in step S3), for the text description of a given action sequence fragment, a cross-modal attention matrix is used to weight the visual features to integrate relevant multimodal context information, namely:

[0018]

[0019] Where, It is the text embedding feature of multimodal fusion after weighted optimization; through the learned cross-modal attention matrix For the matrix V v Weighted and fused text features F t To calculate the multimodal enhanced expression, the multimodal information is integrated through cross-modal attention interaction learning, which enhances the semantic expression of the motion sequence.

[0020] Preferably, in step S4), the cross-modal semantic alignment using a self-supervisory approach comprises the following steps:

[0021] S411) Using the Drop-DTW method to identify the correspondence between the visual sub-action features and the multimodal fused text embedding features, the matched visual sub-action features and the text embedding features are formed into positive cross-modal representation pairs; unmatched cross-modal representation pairs are used as negative samples for training the regression model; that is:

[0022]

[0023] Where, represents the cross-modal representation matrix, representing the visual sub-action features With text embedding features The matching results; D(·) represents the Drop-DTW method;

[0024] S412), by comparing the loss function Perform self-supervised sub-action parsing to improve the similarity between positive sample pairs and increase the distance between mismatched sample pairs;

[0025]

[0026] Where γ is the temperature parameter; t * is the index of the text modality embedding feature; text embedding feature and visual sub-action features Whether a positive sample pair is formed by a cross-modal representation matrix Decide;

[0027] S413), according to the contrast loss function Get the visual-text alignment loss Right now:

[0028]

[0029] The first term aims to use the visual features of sub-actions as the benchmark to convert the visual sub-action features into And the corresponding text embedding features Alignment; the second item embeds the text into the feature based on the sub-action text feature and visual sub-action features Alignment, ultimately achieving cross-modal semantic alignment of sub-action sequences; S represents the number of sub-actions; N represents the number of action sequences.

[0030] As a preference, in step S4), the global semantic contrast learning loss function for global semantic contrast learning is for:

[0031]

[0032] Where, P v Represents Form an index set of positive sample pairs; M represents the number of sample pairs; γ is the temperature parameter; N v is the index set of negative sample pairs.

[0033] As a preference, in step S4), in order to optimize the training process and incorporate prior knowledge, a diversity regularization loss is introduced. Regularization loss through diversity The sub-action sequences parsed from the video sequence are encouraged to have low cosine similarity, i.e.:

[0034]

[0035] Where S represents the total number of sub-actions; i, j represent sub-action index variables, represents the i-th enhanced visual feature.

[0036] As a preference, in step S5), for the visual sub-action feature The regression model first uses a multi-layer perceptron MLP to Mapped to a random score variable y following a Gaussian distribution v ;Right now:

[0037]

[0038] Where, Represents the vector after MLP mapping; Indicates fitting a vector with a Gaussian distribution distribution; σ is the variance; μ is the mean; the mean and variance are used to quantify the average value and uncertainty of the evaluation score, respectively.

[0039] Preferably, in step S5), the predicted score is output by sampling from the score distribution using a reparameterization method. Right now:

[0040]

[0041] Where ε~N(0,1) is a standard normal random variable.

[0042] Preferably, in step S5), in order to supervise the regression of the score distribution, the mean square error MSE is used to define the regression loss Right now:

[0043]

[0044] Where N represents the number of action sequences; y v (i) represents the true score corresponding to the i-th motion subsequence; represents the prediction score corresponding to the i-th motion subsequence;

[0045] Therefore, the total loss function for:

[0046]

[0047] in, is the regression loss, is the diversity regularization loss, is the visual-text alignment loss, is the global semantic contrastive learning loss.

[0048] Preferably, in step S5), the difference between the predicted score and the true score is measured by using the Spearman rank correlation coefficient ρ, that is:

[0049]

[0050] Where y v and denote the true value and predicted score of quality assessment respectively.

[0051] Preferably, the present invention further provides a cross-modal temporal semantic alignment action quality assessment system, comprising:

[0052] Feature extraction module, used to extract visual features F from motion sequence video and motion sequence text description respectively v and text features Ft ;

[0053] Sub-action parsing module, which is used to learn the visual features F through self-attention v Enhance to obtain enhanced visual sub-action features

[0054] The cross-modal attention interaction learning module is used to capture the semantic expression between video sequences and text descriptions and align cross-modal feature expressions to integrate the high-level semantic information of action sequences. By learning the cross-modal attention matrix and using it to weight visual features to integrate relevant multimodal contextual information, the weighted optimized multimodal fusion text embedding feature is obtained.

[0055] A cross-modal semantic alignment module, which performs cross-modal semantic alignment in a self-supervised manner and uses matching relationships to supervise the global semantic contrastive learning of other visual-text pairs;

[0056] The action quality assessment module is used to evaluate the quality of actions based on the score distribution regression method.

[0057] As an example, the feature extraction module extracts the feature from the motion sequence X through the visual feature extraction network P3D. v Extract visual features F v ; and extract text features F from the motion sequence text description through the text feature extraction network BERT t .

[0058] As a preference, the sub-action parsing module adopts a self-attention encoder to aggregate the temporal feature expression of the action sequence, so that the model can learn the global context-related features of the action sequence to enhance the segment-level feature representation. The aggregated weights are learned through the correlation between segments; and a multi-layer Transformer decoder is used as the baseline network for self-attention calculation, setting S learnable query The memory vector is composed of the visual features F v and its projection matrix V v Act as, perform self-attention calculation; for a given T v action sequence fragments, using the learned self-attention matrix to focus on the visual features F v Perform enhancements, namely:

[0059]

[0060] Where, represents the enhanced visual sub-action feature; T v Indicates the number of sequence segments in the video; represents the self-attention weight matrix; V v is the visual feature Fv The projection matrix.

[0061] As an example, the cross-modal attention interaction learning module is based on the text feature F t The projection matrix Q t Perform text context queries As the query vector input of the cross attention mechanism, the visual features F v Acts as a memory; this integration can optimize the representation of visual features by introducing relevant text and its contextual key descriptions. For a given action sequence segment T t The text description of , uses a cross-modal attention matrix to weight the visual features to integrate relevant multimodal context information, namely:

[0062]

[0063] Where, It is the text embedding feature of multimodal fusion after weighted optimization; through the learned cross-modal attention matrix For the matrix V v Weighted and fused text features F t To calculate the multimodal enhanced expression, the multimodal information is integrated through cross-modal attention interaction learning, which enhances the semantic expression of the motion sequence.

[0064] Preferably, the cross-modal semantic alignment module uses the Drop-DTW method to identify the correspondence between visual sub-action features and multimodal fused text embedding features, and forms a positive cross-modal representation pair with the matched visual sub-action features and text embedding features; the unmatched cross-modal representation pairs are used as negative samples for training the regression model; that is:

[0065]

[0066] Where, represents the cross-modal representation matrix, representing the visual sub-action features With text embedding features The matching results; D(·) represents the Drop-DTW method;

[0067] Then, by contrasting the loss function Perform self-supervised sub-action parsing to improve the similarity between positive sample pairs and increase the distance between mismatched sample pairs;

[0068]

[0069] Where γ is the temperature parameter; t * is the index of the text modality embedding feature; text embedding feature and visual sub-action features Whether a positive sample pair is formed by a cross-modal representation matrix Decide;

[0070] And according to the contrast loss function Get the visual-text alignment loss Full visual-text alignment loss It is defined as a combination of two Info-NCE contrastive losses, one for sub-action feature comparison and the other for text semantic embedding comparison. The visual-text alignment loss is defined as follows:

[0071]

[0072] The first term aims to use the visual features of sub-actions as the benchmark to convert the visual sub-action features into And the corresponding text embedding features Alignment; the second item embeds the text into the feature based on the sub-action text feature and visual sub-action features Alignment, ultimately achieving cross-modal semantic alignment of sub-action sequences; S represents the number of sub-actions; N represents the number of action sequences.

[0073] As a preference, the cross-modal semantic alignment module performs global semantic contrast learning on the global semantic contrast learning loss function for:

[0074]

[0075] Where, P v Represents Form an index set of positive sample pairs; M represents the number of samples; γ is the temperature parameter; N v is the index set of negative sample pairs.

[0076] As a preference, the cross-modal semantic alignment module introduces a diversity regularization loss To encourage the sub-action sequences parsed from the video sequence to have low cosine similarity, that is:

[0077]

[0078] Where S represents the total number of sub-actions; i and j represent sub-action index variables.

[0079] As an example, the action quality assessment module first uses a multi-layer perceptron MLP to Mapped to a random score variable y following a Gaussian distribution v ;Right now:

[0080]

[0081]

[0082] Where, Represents the vector after MLP mapping; Indicates fitting a vector with a Gaussian distribution distribution; σ is the variance; μ is the mean; the mean and variance are used to quantify the average value and uncertainty of the evaluation score, respectively;

[0083] And by sampling from the score distribution using the reparameterization method, the predicted score is output Right now:

[0084]

[0085] Where ε~N(0,1) is a standard normal random variable.

[0086] Preferably, the action quality assessment module measures the difference between the predicted score and the true score by using the Spearman rank correlation coefficient ρ, that is:

[0087]

[0088] Where y v and denote the true value and predicted score of quality assessment respectively.

[0089] The beneficial effects of the present invention are:

[0090] 1. This paper enhances the high-level semantic representation of sub-action sequences by mining the alignment relationship between different modalities in video sequences and the temporal correlation within the sequences, thereby obtaining accurate motion sequence quality assessment results;

[0091] 2. This invention promotes interaction between different modalities by capturing semantic alignment of actions between video sequences and text descriptions, helping to improve the accuracy of semantic definitions of action subsequences. It also obtains high-level semantic definitions of sub-actions through self-supervised learning, thereby parsing motion sequences into semantically aligned sub-action sequences.

[0092] 3. The present invention further improves the accuracy and interpretability of score prediction in motion quality assessment by performing action semantic verification and action specification evaluation on the sub-action sequences after motion sequence parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0094] Figure 2 A schematic diagram of a process framework of an embodiment of the present invention;

[0095] Figure 3 Schematic diagram of a self-supervised sub-action semantic parsing module for a motion sequence according to an embodiment of the present invention;

[0096] Figure 4 Schematic diagram of the framework of the system of the embodiment of the present invention. DETAILED DESCRIPTION

[0097] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0098] like Figure 1-3 As shown, this embodiment provides a method for evaluating action quality based on cross-modal temporal semantic alignment, including the following steps:

[0099] S1) Obtain the user's motion sequence video data; extract visual features F from the motion sequence video and the motion sequence text description respectively v and text features F t ;

[0100] In this embodiment, the user's motion sequence data X is obtained. v , where X v ={x v ,y v ; v = 1, ..., N}; is the video sequence; T, h, W, C represent the length, height, width and number of feature channels of the video sequence respectively; y v Score label; N represents the number of action sequences.

[0101] This embodiment uses motion sequence data X v For regression model R θ (·) is trained, and the regression model R θ (·) For the input motion sequence data X v Perform action quality assessment and predict target scores through regression Right now:

[0102]

[0103] Where, Learning networks for features;

[0104] In addition, this embodiment extracts the feature from the motion sequence data X by the visual feature extraction network P3D. v Extract visual features F v ; Extract text features F from the motion sequence text description through the text feature extraction network BERT t ,in Among them, T t Represents the number of text segments and D represents the dimension of the learned features.

[0105] S2), visual feature F v Perform self-supervised sub-action analysis to obtain enhanced visual sub-action features; Figure 2 and 3 As shown, the details are as follows:

[0106] The regression model uses a self-attention encoder to aggregate the temporal feature expression of the action sequence, so that the model can learn the global context-related features of the action sequence to enhance the segment-level feature representation. The weight of the aggregation is learned through the correlation between the segments. The self-attention calculation is performed by using a multi-layer Transformer decoder as the baseline network, and S learnable query The memory vector is composed of the visual features F v and its projection matrix V v In this way, the action sequence can be parsed and divided into independent sub-actions, thereby obtaining sub-action expressions with high-level semantics and capturing the temporal relationship between fine-grained sub-actions.

[0107] Wherein, the visual feature F v The projection learning process is achieved through learnable weight parameters The input visual feature F v Projection to matrix V v The specific process is expressed as:

[0108]

[0109] For a given T v action sequence fragments, using the learned self-attention matrix to focus on the visual features F v Perform enhancements, namely:

[0110]

[0111] Where, represents the enhanced visual sub-action features; is a series of enhanced context vectors, and the vectors in the sequence have certain distinguishability; each vector is defined as a sub-action; T v Indicates the number of sequence segments in the video; represents the self-attention weight matrix; V v is the visual feature F v The projection matrix.

[0112] The self-attention weight The calculation expression is:

[0113]

[0114] Where Qv is the query matrix; τ is a learnable temperature parameter; the attention value is controlled by the temperature parameter τ and is used to enhance the motion sequence features to make the visual features more discriminative; Represents a transpose operation.

[0115] S3), visual feature F v and text features F t Conduct cross-modal attention interaction learning; the details are as follows:

[0116] In the cross-modal attention interaction learning process, the text feature F t The projection matrix Q t Used as a text context query As the query vector input of the cross attention mechanism, the visual features F v Acting as a memory; this integration enables the regression model to optimize the representation of visual features by incorporating relevant text and its contextual key descriptions.

[0117] For a given action sequence segment T t The text description of , uses a cross-modal attention matrix to weight the visual features to integrate relevant multimodal context information, namely:

[0118]

[0119] Where, It is the text embedding feature of multimodal fusion after weighted optimization; through the learned cross-modal attention matrix For the matrix V v Weighted and fused text features F t To calculate multimodal enhanced expression, we integrate multimodal information through cross-modal attention interaction learning to enhance the semantic expression of motion sequences.

[0120] The cross-modal attention mechanism utilizes the interaction between visual and textual features to optimize the visual representation. The cross-modal attention matrix The expression is:

[0121]

[0122] Where, F v is the visual feature; Q t is the text feature F t The projection matrix V v is the visual feature F v The projection matrix of ; τ is a learnable temperature parameter; T represents the transpose operation.

[0123] S4) Use self-supervision to perform cross-modal semantic alignment and global semantic contrast learning; the details are as follows:

[0124] Cross-modal semantic alignment further bridges the gap between visual content and textual representations, enabling the model to establish stronger associations between visual sub-action features and their corresponding textual descriptions. These textual descriptions, as important supplementary information, significantly enhance the model's ability to learn more discriminative sub-action representations, which is crucial for effectively improving sub-action parsing accuracy and motion quality assessment.

[0125] The cross-modal semantic alignment using self-supervision includes the following steps:

[0126] S411) Using the Drop-DTW method to identify the correspondence between the visual sub-action features and the multimodal fused text embedding features, the method automatically detects and removes outliers in the sequence, thereby filtering out irrelevant information between the visual and text modalities; matching visual sub-action features and text embedding features form positive cross-modal representation pairs; unmatched cross-modal representation pairs are used as negative samples for training the regression model; that is:

[0127]

[0128] Where, represents the cross-modal representation matrix, representing the visual sub-action features With text embedding features The matching results; D(·) represents the Drop-DTW method;

[0129] S412), by comparing the loss function Perform self-supervised sub-action parsing to improve the similarity between positive sample pairs and increase the distance between mismatched sample pairs;

[0130]

[0131] Where γ is the temperature parameter; t * is the index of the text modality embedding feature; text embedding feature and visual sub-action features Whether a positive sample pair is formed by a cross-modal representation matrix Decide;

[0132] S413), according to the contrast loss function Get the visual-text alignment loss Right now:

[0133]

[0134] The first term aims to use the visual features of sub-actions as the benchmark to convert the visual sub-action features into And the corresponding text embedding features Alignment; the second item embeds the text into the feature based on the sub-action text feature and visual sub-action features Alignment, ultimately achieving cross-modal semantic alignment of sub-action sequences; S represents the number of sub-actions; N represents the number of action sequences.

[0135] In this embodiment, the global semantic contrast loss is used to extract text features and sub-action features from different videos to create negative sample pairs, thereby extending the generalization of the model. for:

[0136]

[0137] Where, P v Represents Form an index set of positive sample pairs; M represents the number of samples; γ is the temperature parameter; N v is the index set of negative sample pairs.

[0138] And in order to optimize the training process and incorporate prior knowledge, the diversity regularization loss is introduced Regularization loss through diversity The sub-action sequences parsed from the video sequence are encouraged to have low cosine similarity, i.e.:

[0139]

[0140] Where S represents the total number of sub-actions; i and j represent sub-action index variables.

[0141] This loss enhances the model's generalization ability, enabling it to accurately parse sub-actions in complex scenarios. Therefore, this method promotes the diversity of sub-action sequences and improves the overall performance of AQA by removing duplicate text action features.

[0142] S5) Use a method based on score distribution regression to evaluate the quality of the action; the details are as follows:

[0143] For visual sub-action features The regression model first uses a multi-layer perceptron MLP to Mapped to a random score variable y following a Gaussian distribution v ;Right now:

[0144]

[0145] Where, Represents the vector after MLP mapping; Indicates fitting a vector with a Gaussian distribution distribution; σ is the variance; μ is the mean; the mean and variance are used to quantify the average value and uncertainty of the evaluation score, respectively.

[0146] Output the predicted score by sampling from the score distribution using a reparameterization method Right now:

[0147]

[0148] Where ε~N(0,1) is a standard normal random variable.

[0149] In this embodiment, in order to supervise the regression of the score distribution, the mean square error MSE is used to define the regression loss Right now:

[0150]

[0151] Where N represents the number of action sequences; y v (i) represents the true score corresponding to the i-th motion subsequence; represents the prediction score corresponding to the i-th motion subsequence;

[0152] Therefore, the total loss function for:

[0153]

[0154] in, is the regression loss, is the diversity regularization loss, is the visual-text alignment loss, is the global semantic contrastive learning loss.

[0155] Preferably, in step S5), the difference between the predicted score and the true score is measured by using the Spearman rank correlation coefficient ρ, and the value of the Spearman rank correlation coefficient ρ is -1 to 1. The larger the value, the smaller the difference between the predicted and true scores, that is:

[0156]

[0157] Where y v and denote the true value and predicted score of quality assessment respectively.

[0158] Example 2

[0159] like Figure 2-4 As shown, this embodiment provides a cross-modal temporal semantic alignment action quality assessment system, including:

[0160] Feature extraction module, used to extract visual features F from motion sequence video and motion sequence text description respectively v and text features F t ;

[0161] Sub-action parsing module, which is used to learn the visual features F through self-attention v Enhance to obtain enhanced visual sub-action features

[0162] The cross-modal attention interaction learning module is used to capture the semantic expression between video sequences and text descriptions and align cross-modal feature expressions to integrate the high-level semantic information of action sequences. By learning the cross-modal attention matrix and using it to weight visual features to integrate relevant multimodal contextual information, the weighted optimized multimodal fusion text embedding feature is obtained.

[0163] A cross-modal semantic alignment module, which performs cross-modal semantic alignment in a self-supervised manner and uses matching relationships to supervise the global semantic contrastive learning of other visual-text pairs;

[0164] The action quality assessment module is used to evaluate the quality of actions based on the score distribution regression method.

[0165] As preferred in this embodiment, the feature extraction module extracts the feature from the motion sequence X through the visual feature extraction network P3D. v Extract visual features F v ; and extract text features F from the motion sequence text description through the text feature extraction network BERT t .

[0166] As preferred in this embodiment, the sub-action parsing module adopts a self-attention encoder to aggregate the temporal feature expression of the action sequence, so that the model can learn the global context-related features of the action sequence to enhance the segment-level feature representation. The aggregated weights are learned through the correlation between segments; and a multi-layer Transformer decoder is used as the baseline network for self-attention calculation, setting S learnable query

[0167] The memory vector is composed of the visual features F v and its projection matrix V v Act as, perform self-attention calculation; for a given T v action sequence fragments, using the learned self-attention matrix to focus on the visual features F v Perform enhancements, namely:

[0168]

[0169] Where, represents the enhanced visual sub-action features; represents the self-attention weight matrix; V v is the visual feature F v The projection matrix.

[0170] Wherein, the visual feature F v The projection learning process is achieved through learnable weight parameters The input visual feature F v Projection to matrix V v The specific process is expressed as:

[0171]

[0172] The self-attention weight The calculation expression is:

[0173]

[0174] Where Q v is the query matrix; τ is a learnable temperature parameter; the attention value is controlled by the temperature parameter τ and is used to enhance the motion sequence features to make the visual features more discriminative; Represents a transpose operation.

[0175] As a preferred embodiment of this invention, the cross-modal attention interaction learning module is based on the text feature F t The projection matrix Q t Perform text context queries As the query vector input of the cross attention mechanism, the visual features F v Acts as a memory; this integration can optimize the representation of visual features by introducing relevant text and its contextual key descriptions. For a given action sequence segment T t The text description of , uses a cross-modal attention matrix to weight the visual features to integrate relevant multimodal context information, namely:

[0176]

[0177] Where, It is the text embedding feature of multimodal fusion after weighted optimization; through the learned cross-modal attention matrix For the matrix V v Weighted and fused text features F t To calculate the multimodal enhanced expression, the multimodal information is integrated through cross-modal attention interaction learning, which enhances the semantic expression of the motion sequence.

[0178] Among them, the cross-modal attention mechanism uses the interaction between visual and text features to optimize the visual representation. The cross-modal attention matrix The expression is:

[0179]

[0180] Where, F v is the visual feature; Q t is the text feature F t The projection matrix V v is the visual feature F v The projection matrix of ; τ is the learnable temperature parameter; Represents a transpose operation.

[0181] As a preferred embodiment of this invention, the cross-modal semantic alignment module uses the Drop-DTW method to identify the correspondence between the visual sub-action features and the multimodal fused text embedding features, and forms a positive cross-modal representation pair with the matched visual sub-action features and the text embedding features; the unmatched cross-modal representation pairs are used as negative samples for training the regression model; that is:

[0182]

[0183] Where, represents the cross-modal representation matrix, representing the visual sub-action features With text embedding features The matching results; D(·) represents the Drop-DTW method;

[0184] Then, by contrasting the loss function Perform self-supervised sub-action parsing to improve the similarity between positive sample pairs and increase the distance between mismatched sample pairs;

[0185]

[0186] Where γ is the temperature parameter; t * is the index of the text modality embedding feature; text embedding feature and visual sub-action features Whether a positive sample pair is formed by a cross-modal representation matrix Decide;

[0187] And according to the contrast loss function Get the visual-text alignment loss Right now:

[0188]

[0189] The first term aims to use the visual features of sub-actions as the benchmark to convert the visual sub-action features into And the corresponding text embedding features Alignment; the second item embeds the text into the feature based on the sub-action text feature and visual sub-action features Alignment, ultimately achieving cross-modal semantic alignment of sub-action sequences; S represents the number of sub-actions; N represents the number of action sequences.

[0190] As preferred in this embodiment, the global semantic contrast learning loss function of the cross-modal semantic alignment module for global semantic contrast learning is for:

[0191]

[0192] Where S represents the total number of sub-actions; P v Represents Form an index set of positive sample pairs; M represents the number of samples; γ is the temperature parameter; N v is the index set of negative sample pairs.

[0193] As a preferred embodiment of this invention, the cross-modal semantic alignment module introduces a diversity regularization loss. To encourage the sub-action sequences parsed from the video sequence to have low cosine similarity, that is:

[0194]

[0195] Where S represents the total number of sub-actions; i, j represent sub-action index variables; represents the i-th enhanced visual feature, and the subscript v represents the visual feature.

[0196] As preferred in this embodiment, the action quality assessment module first uses a multi-layer perceptron MLP to Mapped to a random score variable y following a Gaussian distribution v ;Right now:

[0197]

[0198] Where, Represents the vector after MLP mapping; Indicates fitting a vector with a Gaussian distribution distribution; σ is the variance; μ is the mean; the mean and variance are used to quantify the average value and uncertainty of the evaluation score, respectively;

[0199] And by sampling from the score distribution using the reparameterization method, the predicted score is output Right now:

[0200]

[0201] Where ε~N(0,1) is a standard normal random variable.

[0202] As a preferred embodiment of the present invention, the action quality assessment module measures the difference between the predicted score and the true score by using the Spearman rank correlation coefficient ρ, that is:

[0203]

[0204] Where y v and denote the true value and predicted score of quality assessment respectively.

[0205] And in order to supervise the regression of the score distribution, the mean square error MSE is used to define the regression loss Right now:

[0206]

[0207] Where N represents the number of action sequences; y v (i) represents the true score corresponding to the i-th motion subsequence; represents the prediction score corresponding to the i-th motion subsequence;

[0208] Therefore, the total loss function for:

[0209]

[0210] in, is the regression loss, is the diversity regularization loss, is the visual-text alignment loss, is the global semantic contrastive learning loss.

[0211] The above embodiments and descriptions are only for explaining the principles and best embodiments of the present invention. Without departing from the spirit and scope of the present invention, the present invention may be subject to various changes and improvements, which shall fall within the scope of the invention to be protected.

Claims

1. A method for action quality assessment based on cross-modal temporal semantic alignment, characterized by: The steps include: S1) Obtain the user's motion sequence video data; and extract visual features F from the motion sequence video and the motion sequence text description respectively. v and text features F t ; S2), visual feature F v Perform self-supervised sub-action parsing to obtain enhanced visual sub-action features; S3), visual feature F v and text features F t Conduct cross-modal attention interaction learning; S4) Use self-supervision to perform cross-modal semantic alignment and global semantic contrast learning; S5) Use the method based on score distribution regression to evaluate the quality of the action.

2. The method for assessing motion quality of cross-modal temporal semantic alignment according to claim 1, characterized in that: In step S1), the visual feature extraction network P3D is used to extract the motion sequence X v Extract visual features F v ; Extract text features F from the motion sequence text description through the text feature extraction network BERT t .

3. The method for assessing motion quality of cross-modal temporal semantic alignment according to claim 2, characterized in that: In step S2), a self-attention encoder is used to aggregate the temporal feature expression of the action sequence, so that the model can learn the global context-related features of the action sequence to enhance the segment-level feature representation. The weight of the aggregation is learned through the correlation between the segments; and a multi-layer Transformer decoder is used as the baseline network for self-attention calculation, setting S learnable query The memory vector is composed of the visual features F v and its projection matrix V v Act as,and perform self-attention calculation.

4. The method for assessing motion quality of cross-modal temporal semantic alignment according to claim 3, characterized in that: In step S2), the v The video of the sequence clips is processed by using the learned self-attention matrix to focus on the visual features F v Perform enhancements, namely: Where, represents the enhanced visual sub-action feature; T v Indicates the number of action segments in the video sequence; represents the self-attention weight matrix; V v is the visual feature F v The projection matrix; The visual feature F v The projection learning process is achieved through learnable weight parameters The input visual feature F v Projection to matrix V v The specific process is expressed as: The self-attention weight The calculation expression is: Where Q v is the query matrix; τ is a learnable temperature parameter; the attention value is controlled by the temperature parameter τ and is used to enhance the motion sequence features to make the visual features more discriminative; Represents a transpose operation.

5. The method for motion quality assessment of cross-modal temporal semantic alignment according to claim 4, characterized in that: In step S3), during the cross-modal attention interaction learning process, the text feature F t The projection matrix is used as a textual context query, as the query vector input of the cross attention mechanism, and the visual feature F v Acting as a memory; this integration enables the regression model to optimize the representation of visual features by incorporating relevant text and its contextual key descriptions.

6. The method for assessing motion quality of cross-modal temporal semantic alignment according to claim 5, characterized in that: In step S3), for the text description of a given action sequence segment, a cross-modal attention matrix is used to weight the visual features to integrate relevant multimodal context information, namely: Where, It is the text embedding feature after weighted optimization and multimodal fusion; Through the learned cross-modal attention matrix For the matrix V v Weighted and fused text features F t To calculate multimodal enhanced expression, we integrate multimodal information through cross-modal attention interaction learning, thus enhancing the semantic expression of motion sequences. Among them, the cross-modal attention mechanism uses the interaction between visual and text features to optimize the visual representation. The cross-modal attention matrix The expression is: Where, F v is the visual feature; Q t is the text feature F t The projection matrix V v is the visual feature F v The projection matrix of ; τ is a learnable temperature parameter; T represents the transpose operation.

7. The method for assessing motion quality of cross-modal temporal semantic alignment according to claim 6, characterized in that: In step S4), the cross-modal semantic alignment using a self-supervised approach includes the following steps: S411) Using the Drop-DTW method to identify the correspondence between the visual sub-action features and the multimodal fused text embedding features, the matched visual sub-action features and the text embedding features are formed into positive cross-modal representation pairs; unmatched cross-modal representation pairs are used as negative samples for training the regression model; that is: Where, represents the cross-modal representation matrix, representing the visual sub-action features With text embedding features The matching results; D(·) represents the Drop-DTW method; S412), by comparing the loss function Perform self-supervised sub-action parsing, the contrast loss function Improve the similarity between positive sample pairs and increase the distance between mismatched sample pairs; Where γ is the temperature parameter; t * is the index of the text modality embedding feature; text embedding feature and visual sub-action features Whether a positive sample pair is formed by a cross-modal representation matrix Decide; S413), complete visual text alignment loss It is defined as a combination of two Info-NCE contrastive losses, one for sub-action feature comparison and the other for text semantic embedding comparison. The visual-text alignment loss is defined as follows: The first term aims to use the visual features of sub-actions as the benchmark to convert the visual sub-action features into And the corresponding text embedding features Alignment; the second item embeds the text into the feature based on the sub-action text feature and visual sub-action features Alignment, ultimately achieving cross-modal semantic alignment of sub-action sequences; S represents the number of sub-actions; N represents the number of action sequences.

8. The method for motion quality assessment of cross-modal temporal semantic alignment according to claim 7, characterized in that: In step S4), the loss function for global semantic contrast learning is for: Where, P v Represents Form an index set of positive sample pairs; M represents the number of sample pairs; γ is the temperature parameter; N v is the index set of negative sample pairs; By introducing diversity regularization loss Encourage diversity among sub-actions, i.e.: Where S represents the number of sub-actions; i, j represent sub-action index variables, represents the i-th enhanced visual feature.

9. The method for motion quality assessment of cross-modal temporal semantic alignment according to claim 8, characterized in that: In step S5), for the visual sub-action feature The regression model first uses a multi-layer perceptron MLP to Mapped to a random score variable y following a Gaussian distribution v ;Right now: Where, Represents the vector after MLP mapping; Indicates fitting a vector with a Gaussian distribution distribution; σ is the variance; μ is the mean; the mean and variance are used to quantify the average value and uncertainty of the evaluation score, respectively; And by sampling from the score distribution using the reparameterization method, the predicted score is output Right now: Where ε~N(0,1) is a standard normal random variable; The difference between the predicted score and the true score is then measured by using the Spearman rank correlation coefficient ρ, which is: Where y v and denote the true value and predicted score of quality assessment respectively.

10. A cross-modal temporal semantic alignment action quality assessment system, characterized by: The system uses the method according to any one of claims 1 to 9 to perform action quality assessment, and the system includes: Feature extraction module, used to extract visual features F from motion sequence video and motion sequence text description respectively v and text features F t ; Sub-action parsing module, which is used to learn the visual features F through self-attention v Enhance to obtain enhanced visual sub-action features The cross-modal attention interaction learning module is used to capture the semantic expression between video sequences and text descriptions and align cross-modal feature expressions to integrate the high-level semantic information of action sequences. By learning the cross-modal attention matrix and using it to weight visual features to integrate relevant multimodal contextual information, the weighted optimized multimodal fusion text embedding feature is obtained. A cross-modal semantic alignment module, which performs cross-modal semantic alignment in a self-supervised manner and uses matching relationships to supervise the global semantic contrastive learning of other visual-text pairs; The action quality assessment module is used to evaluate the quality of actions based on the score distribution regression method.

Images