Multi-modal action evaluation model training method and device, equipment and medium

Abstract

The embodiment of the invention provides a multi-modal action evaluation model training method and device, equipment and a medium. The method comprises the following steps: acquiring a video data set, and determining an action step information set according to the video data set; wherein the video data set comprises videos and label information corresponding to the videos, the label information comprises action classification labels and action quality labels, and the action step information set comprises the action classification labels and corresponding action step information; for each video in the video data set, determining thinking chain interpretation information corresponding to the video according to the action step information set; according to the video and the corresponding label information and thinking chain interpretation information, training the initial model to obtain a trained multi-modal motion evaluation model; wherein the multi-modal action evaluation model is used for evaluating classification and quality of actions in the video and giving interpretable analysis. The method is used for achieving the effect of improving motion quality evaluation precision and interpretability.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method, apparatus, device and medium for a multimodal motion evaluation model. Background Technology

[0002] In real-world scenarios such as industrial production, health and recreation, and martial arts, assessing the standardization of movement execution is crucial. For example, a standard movement ensures training effectiveness, prevents sports injuries, and guarantees workplace safety. Movement assessment technology can provide refined feedback and guidance in areas such as personalized fitness instruction, professional skills training, and medical rehabilitation.

[0003] Existing technologies primarily rely on regression tasks, focusing on providing numerical scores or coarse-grained labels for action proficiency. Regression models are trained to assess action quality, i.e., the degree of standardization in action execution. However, the evaluation criteria of existing methods are relatively fixed, making it difficult to meet users' demands for precision in action quality assessment.

[0004] Therefore, there is an urgent need for a solution that can improve the accuracy of motion quality assessment. Summary of the Invention

[0005] The training method, apparatus, equipment, and medium for the multimodal motion evaluation model provided in this application significantly improve the accuracy and interpretability of motion quality evaluation through the "video-text-thought chain" multimodal training paradigm.

[0006] In a first aspect, embodiments of this application provide a training method for a multimodal action evaluation model, comprising:

[0007] Obtain a video dataset and determine the action step information set based on the video dataset. The video dataset includes videos and their corresponding label information. The label information includes action classification labels and action quality labels. The action step information set includes action classification labels and their corresponding action step information. The action step information represents the action technical steps under the action category and is used to structurally describe the execution flow of standard actions.

[0008] For each video in the video dataset, the thought chain explanation information corresponding to the video is determined based on the action step information set; wherein, the thought chain explanation information is an interpretable textual description of the action phenomenon in the video, including observations of the action execution process, error analysis and improvement suggestions;

[0009] Based on the video and its corresponding tag information and thought chain explanation information, the initial multimodal model is trained to obtain the trained multimodal action evaluation model; the multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

[0010] In an optional example, based on the video dataset, a set of motion step information is determined, including:

[0011] For each action category label in the video dataset, determine the action name corresponding to the action category label;

[0012] Input the preset first prompt and action name into the preset first large model to obtain the output action step information; wherein, the preset first large model is used to output action step information;

[0013] Each action category label and its corresponding action step information are defined as an action step information set.

[0014] In an optional example, based on the action step information set, the thought chain explanation information corresponding to the video is determined, including:

[0015] Based on the action category tags corresponding to the video, determine the action step information corresponding to the video from the action step information set;

[0016] The preset second prompt, video, and corresponding action step information and action quality labels are input into the preset second large model to obtain the thought chain explanation information corresponding to the output video; the preset second large model is used to output the thought chain explanation information to realize multimodal comparative analysis.

[0017] In an optional example, the initial model includes a first submodule, a second submodule, and a third submodule; based on the video and corresponding label information and thought chain explanation information, the initial model is trained to obtain a trained multimodal action evaluation model, including:

[0018] The video features are determined as the basis for multimodal input; among them, the video features characterize the semantic representation of the video in the spatiotemporal dimension.

[0019] The video features are input into the first submodule to obtain the output predicted action classification, and the video features are input into the second submodule to obtain the output predicted action quality. Here, the predicted action classification represents the prediction result of the action classification information of the video, and the predicted action quality represents the confidence level of whether the action is standard or not.

[0020] Based on video features and predicted action classification, fusion features are determined; whereby, fusion features characterize the fusion result of video semantics and action step semantics;

[0021] The fused features are input into the third submodule to obtain the output prediction thought chain interpretation;

[0022] The initial model is trained based on the predicted action classification, predicted action quality, predicted thought chain interpretation, label information, and thought chain interpretation information to obtain a trained multimodal action evaluation model.

[0023] In an optional example, the fused features are determined based on video features and predicted action classification, including:

[0024] Based on the predicted action classification, the predicted action steps corresponding to the video are determined from the action step information set; wherein, the predicted action steps represent the action step information corresponding to the predicted action classification.

[0025] Based on the predicted action steps, text features are determined; whereby text features represent the semantic vector of the predicted action steps.

[0026] By fusing video features and text features, a fused feature is obtained, achieving multimodal semantic alignment and complementarity.

[0027] In an optional example, the initial model includes a first cross-attention module, a second cross-attention module, and a convolutional module; video features and text features are fused to obtain fused features, including:

[0028] Using text features as queries and video features as keys and values, text enhancement features are determined based on the first cross-attention module;

[0029] Using video features as queries and text features as keys and values, the video enhancement features are determined based on the second cross-attention module;

[0030] Based on text enhancement features and video enhancement features, fusion features are determined using convolutional modules.

[0031] In an optional example, the initial model is trained based on predicted action classification, predicted action quality, predicted thought chain interpretation, label information, and thought chain interpretation information to obtain a trained multimodal action evaluation model, including:

[0032] The first loss value is determined based on the action classification label in the predicted action classification and label information;

[0033] The second loss value is determined based on the predicted action quality and the action quality label in the label information;

[0034] The third loss value is determined based on the predicted thought chain explanation and the thought chain explanation information;

[0035] The comprehensive loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0036] Based on the comprehensive loss value, the initial model is trained to obtain a trained multimodal action evaluation model.

[0037] Secondly, embodiments of this application provide a training apparatus for a multimodal action evaluation model, comprising:

[0038] The first processing unit is used to acquire a video dataset and determine an action step information set based on the video dataset. The video dataset includes videos and corresponding label information. The label information includes action classification labels and action quality labels. The action step information set includes action classification labels and corresponding action step information. The action step information represents the action technique steps under the action category.

[0039] The second processing unit is used to determine the corresponding thought chain explanation information for each video in the video dataset based on the action step information set; wherein, the thought chain explanation information represents the text description and analysis suggestions of the action phenomenon in the video;

[0040] The training unit is used to train the initial model based on the video and the corresponding label information and thought chain explanation information to obtain the trained multimodal action evaluation model. The multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

[0041] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0042] The memory stores instructions that the computer executes;

[0043] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0044] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0045] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0046] The multimodal motion evaluation model training method, apparatus, device, and medium provided in this application embodiment acquire a video dataset and determine a motion step information set based on the video dataset. Further, for each video in the video dataset, the corresponding thought chain explanation information is determined based on the motion step information set. The initial model is then trained based on the video, its corresponding label information, and the thought chain explanation information to obtain a trained multimodal motion evaluation model. The video dataset includes videos and their corresponding label information. The label information includes motion classification labels and motion quality labels. The motion step information set includes motion classification labels and corresponding motion step information. The motion step information represents the motion technique steps under a motion category. The thought chain explanation information represents the textual description and analysis suggestions of the motion phenomena in the video. The multimodal motion evaluation model is used to evaluate the classification and quality of motions in the video and provide interpretability analysis. This application's method incorporates thought chain interpretation information into model training, helping the model better analyze and deduce the quality of actions in videos. This improves the accuracy of multimodal action evaluation models in assessing action quality. Furthermore, compared to existing models that only provide numerical scores or coarse-grained evaluations of actions in videos, the multimodal action evaluation model can assess the classification and quality of actions in videos and provide interpretable analysis. That is, it can provide quantifiable action standard levels and generate human-readable correction prompts for action evaluation in videos, significantly reducing the false positive rate and increasing user trust. This application's method aims to improve the accuracy of action quality assessment. Attached Figure Description

[0047] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0048] Figure 1 A flowchart illustrating the training method for the multimodal action evaluation model provided in this application. Figure 1 ;

[0049] Figure 2 A flowchart illustrating the training method for the multimodal action evaluation model provided in this application. Figure 2 ;

[0050] Figure 3 A schematic diagram of the architecture of the training method for the multimodal action evaluation model provided in this application;

[0051] Figure 4 A schematic diagram of the structure of the training device for the multimodal motion evaluation model provided in this application;

[0052] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0053] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0056] In real-world scenarios such as industrial production, health and recreation, and martial arts, assessing the standardization of movement execution is crucial. For example, a standard movement ensures training effectiveness, prevents sports injuries, and guarantees workplace safety. Movement assessment technology can provide refined feedback and guidance in areas such as personalized fitness instruction, professional skills training, and medical rehabilitation.

[0057] Existing technologies primarily rely on regression tasks, focusing on providing numerical scores or coarse-grained labels for action proficiency. Regression models are trained to assess action quality, i.e., the degree of standardization in action execution. However, the evaluation criteria of existing methods are relatively fixed, making it difficult to meet users' demands for precision in action quality assessment.

[0058] Specifically, existing methods often lack unified, objective, and machine-readable evaluation criteria, and the generated texts are usually simple conclusive statements.

[0059] Therefore, there is an urgent need for a solution that can improve the accuracy of motion quality assessment.

[0060] The multimodal motion evaluation model training method, apparatus, device, and medium provided in this application embodiment acquire a video dataset and determine a motion step information set based on the video dataset. Further, for each video in the video dataset, the corresponding thought chain explanation information is determined based on the motion step information set. The initial model is then trained based on the video, its corresponding label information, and the thought chain explanation information to obtain a trained multimodal motion evaluation model. The video dataset includes videos and their corresponding label information. The label information includes motion classification labels and motion quality labels. The motion step information set includes motion classification labels and corresponding motion step information. The motion step information represents the motion technique steps under a motion category. The thought chain explanation information represents the textual description and analysis suggestions of the motion phenomena in the video. The multimodal motion evaluation model is used to evaluate the classification and quality of motions in the video and provide interpretability analysis. This application's method incorporates thought chain interpretation information into model training, helping the model better analyze and deduce the quality of actions in videos. This improves the accuracy of multimodal action evaluation models in assessing action quality. Furthermore, compared to existing models that only provide numerical scores or coarse-grained evaluations of actions in videos, the multimodal action evaluation model can assess the classification and quality of actions in videos and provide interpretable analysis. That is, it can provide quantifiable action standard levels and generate human-readable correction prompts for action evaluation in videos, significantly reducing the false positive rate and increasing user trust. This application's method aims to improve the accuracy of action quality assessment.

[0061] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0062] Figure 1 A flowchart illustrating the training method for the multimodal action evaluation model provided in this application. Figure 1 ,like Figure 1 As shown, the method includes:

[0063] S101. Obtain the video dataset and determine the action step information set based on the video dataset; wherein, the video dataset includes videos and the corresponding label information of the videos, the label information includes action classification labels and action quality labels, the action step information set includes action classification labels and the corresponding action step information, and the action step information represents the action technical steps under the action category.

[0064] S102. For each video in the video dataset, determine the corresponding thought chain explanation information based on the action step information set; wherein, the thought chain explanation information represents the textual description and analysis suggestions of the action phenomena in the video.

[0065] S103. Based on the video and the corresponding tag information and thought chain explanation information, train the initial model to obtain the trained multimodal action evaluation model; wherein, the multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

[0066] In step S101, the videos in the video dataset may include, but are not limited to, human movements, etc. The source of the videos may be public datasets, authorized surveillance videos in industrial scenarios, clips uploaded by fitness software users, etc. For example, the training dataset contains 3392 videos.

[0067] Tag information can refer to the metadata information corresponding to each video, including at least motion category tags and motion quality tags.

[0068] Action category tags can refer to discrete numbers or strings for the categories of actions performed in the video, such as "squat", "deadlift", "clean and jerk" and other action categories. For example, the total number of action category tags is 141.

[0069] Action quality labels can refer to binary or graded markers indicating the degree of standardization of action execution. In this embodiment, binary markers can be used as action quality labels, such as 0 representing standard and 1 representing non-standard. In one possible implementation, action quality labels can also be graded markers, such as a scale of 1 to 10. The larger the grade value, the less standard the action execution.

[0070] For example, action classification labels and action quality labels can be obtained through methods such as manual frame-by-frame observation or expert scoring.

[0071] The action step information set can refer to a data table or key-value storage file with action category labels as the primary key and structured text (i.e., action step information) as the value. The action step information can refer to standard technical steps described in natural language. For example, the action step information includes 5 core action steps and 1 general training suggestion, which is used to help determine the thought chain explanation information corresponding to the video.

[0072] In step S102, the thought chain explanation information can refer to the explanation text relating to the current video action execution status. Specifically, it can be a textual description and analysis suggestion of the action phenomenon in the video, that is, it includes not only the observation of the action phenomenon, but also the analysis of the consequences of errors and improvement suggestions. The output of the explanation text is determined by combining the action step information in the action step information set.

[0073] In step S103, the initial model may include a first submodule, a second submodule, and a third submodule. The first submodule is used to output the predicted action classification, the second submodule is used to output the predicted action quality, and the third submodule is used to output the predicted thought chain explanation.

[0074] Specifically, training the initial model can be achieved by updating the model parameters in the initial model based on the degree of difference between the predicted action classification and the action classification label, the degree of difference between the predicted action quality and the action quality label, and the degree of difference between the predicted thought chain interpretation and the thought chain interpretation information, thus obtaining the trained multimodal action evaluation model.

[0075] As is understandable, multimodal motion evaluation models are used to assess the classification and quality of actions in videos and provide interpretability analysis. By obtaining a trained multimodal motion evaluation model, the accuracy of motion quality assessment in videos can be improved.

[0076] The training method for the multimodal motion evaluation model provided in this application considers applying thought chain interpretation information to the model training, helping the model to better analyze and deduce the motion quality in videos, thereby improving the accuracy of the multimodal motion evaluation model in motion quality assessment. The method of this application aims to improve the accuracy of motion quality assessment.

[0077] Figure 2 A flowchart illustrating the training method for the multimodal action evaluation model provided in this application. Figure 2 ,like Figure 2 As shown, in this embodiment... Figure 1 Based on the examples, the training method for the multimodal action evaluation model is described in detail, including:

[0078] S201. Obtain the video dataset and determine the action step information set based on the video dataset; wherein, the video dataset includes videos and the corresponding label information of the videos, the label information includes action classification labels and action quality labels, the action step information set includes action classification labels and the corresponding action step information, and the action step information represents the action technical steps under the action category.

[0079] In one alternative implementation, determining the action step information set based on the video dataset may include:

[0080] For each action category label in the video dataset, determine the action name corresponding to the action category label; input the preset first prompt and action name into the preset first large model to obtain the output action step information; wherein, the preset first large model is used to output action step information; and determine each action category label and the corresponding action step information as an action step information set.

[0081] The action name can refer to a standard action name in a readable form that is easy for humans to understand, such as "squat," "deadlift," or "clean and jerk." In one possible implementation, the action name can be an action category label. In another possible implementation, there can be a one-to-one correspondence between the action category label and the action name, which can be obtained in advance by staff based on relevant data. For example, the action category label is "AAA," and the action name corresponding to this action category label is "squat."

[0082] The first preset large model can refer to a multimodal large model that has been pre-trained on text corpus and has the ability to follow instructions, and the first preset prompt can refer to the text that guides the first preset large model to output the corresponding result.

[0083] For example, the preset first major model may include, but is not limited to, the model Gemini2.5, etc. The preset first prompt may be expressed as "Please generate standard technical steps for this action name, in the following format: 1. List 5 core steps in chronological order; 2. Add 1 training suggestion". By inputting the first prompt and the action name into the first major model, the action step information output by the first major model can be obtained. The action step information has a one-to-one correspondence with the action name or action category label.

[0084] Furthermore, by classifying each action with its corresponding action step information, a set of action step information can be determined. This set of action step information can then be used as a reference for determining the thought process explanation information corresponding to the video in subsequent steps.

[0085] The advantage of this setup is that the action step information automatically generated based on the first major model can serve as a reference benchmark for comparative learning of the subsequent preset second major model, which can significantly improve the accuracy of determining the explanatory information of the thought chain.

[0086] S202. For each video in the video dataset, determine the corresponding thought chain explanation information based on the action step information set; wherein, the thought chain explanation information represents the textual description and analysis suggestions of the action phenomena in the video.

[0087] In one optional implementation, the thought chain explanation information corresponding to the video is determined based on the action step information set, including:

[0088] Based on the action classification tags corresponding to the video, the action step information corresponding to the video is determined from the action step information set; the preset second prompt, video, and corresponding action step information and action quality tags are input into the preset second large model to obtain the output video corresponding to the thought chain explanation information; wherein, the preset second large model is used to output the thought chain explanation information.

[0089] Specifically, based on the action category tag, the action step information corresponding to the action category tag can be retrieved from the action step information set, that is, the action step information corresponding to the video.

[0090] The preset second large model can refer to a multimodal large model that has video and text understanding capabilities and command following capabilities. The preset second prompt can refer to the text that guides the preset second large model to output the corresponding result. Specifically, the preset second prompt can guide the second large model to compare and analyze the actual actions in the video with the standard steps represented by the action step information, and output a thought chain-style text explanation, that is, thought chain explanation information.

[0091] For example, the preset second major model may include, but is not limited to, the model VideoChat, etc. The preset second prompt may be expressed as "Based on the given video, analyze the actual actions in the video and compare them with the standard steps represented by the provided action step information, and output a thought chain-style text explanation in combination with the provided action quality tags". By inputting the second prompt, the video, the action step information and action quality tags corresponding to the video into the second major model, the output thought chain explanation information corresponding to the video can be obtained.

[0092] It is understandable that the first and second pre-set models are different. The first model outputs action step information, while the second model outputs thought chain explanation information. The division of labor is clear, and the adaptability of the first and second models to their respective output content is guaranteed, thereby providing reliable and accurate data input for the training of the subsequent multimodal action evaluation model.

[0093] S203. Determine the video features of the video; wherein, the video features characterize the semantic representation of the video in the spatiotemporal dimension.

[0094] For example, uniform sampling of the video, i.e., sampling at consistent time intervals (e.g., 0.5 milliseconds), yields a frame sequence, which can be represented as follows: , where N is a positive integer, and each frame in the frame sequence represents an image.

[0095] In one possible implementation, the image of each frame in the frame sequence can be cropped, for example, by reducing the size of each frame's image (the number of color channels in the image). Image width Image height) cropped to This ensures consistency in image processing, thereby improving the rate of video feature determination and reducing computational load.

[0096] In one possible implementation, a pre-trained first neural network model can be used to initialize the video encoder, resulting in a target video encoder. Inputting a sequence of video frames into the target video encoder yields the output video features. For example, the pre-trained first neural network model can be a Video SwinTransformer model.

[0097] S204. Input the video features into the first submodule to obtain the output predicted action classification, and input the video features into the second submodule to obtain the output predicted action quality; wherein, the predicted action classification represents the prediction result of the action classification information of the video, and the predicted action quality represents the confidence level of whether the action is standard or not.

[0098] The first submodule is used to output the predicted action classification, and the second submodule is used to output the predicted action quality.

[0099] In one possible implementation, both the first and second submodules can be composed of a multilayer perceptron (MLP) model. For example, the MLP model includes three network layers: an input layer, a hidden layer, and an output layer.

[0100] In one possible implementation, before inputting the video features into the first or second submodule, global average pooling (GAP) can be performed on the video features to obtain processed video features, which can then be input into the first or second submodule. The advantage of this approach is that it preserves the channel-level semantics of the video features while avoiding overfitting of the MLP model, thereby improving the generalization ability of the first and second submodules and ultimately enhancing the accuracy of the subsequent multimodal action evaluation model.

[0101] S205. Based on video features and predicted action classification, determine the fusion features; whereby the fusion features characterize the fusion result of video semantics and action step semantics.

[0102] In one alternative implementation, step S205 may include:

[0103] S2051. Based on the predicted action classification, determine the predicted action steps corresponding to the video from the action step information set; wherein, the predicted action steps represent the action step information corresponding to the predicted action classification.

[0104] The action step information set includes action category labels and the corresponding action step information. The predicted action category is the output of the first submodule, representing the initial model's prediction result for the action category of the video. Based on this predicted action category, the corresponding action step information can be retrieved from the action step information set using this predicted action category as an index.

[0105] S2052. Determine text features based on the predicted action steps; wherein, the text features represent the semantic vector of the predicted action steps.

[0106] In one possible implementation, the text encoder can be initialized using a pre-trained second neural network model to obtain the target text encoder.

[0107] By inputting the predicted action steps into the target text encoder, the output text features can be obtained. For example, the pre-trained second neural network model can be a Bidirectional Encoder Representations from Transformers (BERT) model. In one possible implementation, the BERT model consists of 12 layers of Transformer encoding blocks stacked sequentially.

[0108] S2053. The video features and text features are fused to obtain the fused features.

[0109] In one optional implementation, the initial model includes a first cross-attention module, a second cross-attention module, and a convolutional module; step S2053 may include:

[0110] Using text features as queries and video features as keys and values, text enhancement features are determined based on the first cross-attention module; using video features as queries and text features as keys and values, video enhancement features are determined based on the second cross-attention module; and based on the text enhancement features and video enhancement features, fusion features are determined based on the convolution module.

[0111] First, we will explain the Cross-Attention Module. The Cross-Attention Module can refer to an attention mechanism in which the query vector comes from one modality or sequence, while the key and value vectors come from another modality or sequence. It is specifically designed to achieve alignment and information fusion between two different information sources.

[0112] In this embodiment, the fusion features can be determined based on bidirectional cross-attention interaction (i.e., the processing of the first cross-attention module and the second cross-attention module).

[0113] The first cross-attention module is used to output text-enhanced features using text features as queries and video features as keys and values; the second cross-attention module is used to output video-enhanced features using video features as queries and text features as keys and values.

[0114] For example, to enable text features to focus on key video segments matching specific action steps, text features are used as the query, and video features as the key and value, to obtain text-enhanced features. Text-enhanced features can be represented as:

[0115] ;

[0116] In the above formula, Indicates text enhancement features, This indicates that the first cross-attention module uses a multi-head attention mechanism to execute the first layer of interaction. This represents the query in the first cross-attention module. This represents the key in the first cross-attention module. This represents the value in the first cross-attention module. This represents the learnable coefficient (e.g., 0.1) in the preset first cross-attention module. Represents text features.

[0117] Simultaneously, to ensure that visual features focus on the most relevant description of the current action in the text, video features are used as the query, and text features as the key and value, to obtain video-enhanced features. Video-enhanced features can be represented as:

[0118] ;

[0119] In the above formula, Indicates video enhancement features, This indicates that the second cross-attention module uses a multi-head attention mechanism to execute the second layer of interaction. This refers to the query in the second cross-attention module. This represents the key in the second cross-attention module. This represents the value in the second cross-attention module. This represents the learnable coefficient (e.g., 0.1) in the preset second cross-attention module. Indicates video features.

[0120] Furthermore, concatenating the text enhancement features and video enhancement features yields concatenated features. Processing these concatenated features using a convolutional module results in fused features.

[0121] For example, the convolutional module can be a convolutional layer. In one possible implementation, the concatenated features are processed sequentially through convolutional layers and batch normalization to obtain fused features. The fused features can be represented as:

[0122] ;

[0123] In the above formula, Indicates fusion characteristics, Indicates text enhancement features, Indicates video enhancement features, This indicates that text enhancement features and video enhancement features are concatenated to obtain concatenated features. Indicates a convolutional layer. (Batch Normalization) refers to the batch normalization operation, which can refer to the standardization of the output of the convolutional layer along the batch and channel dimensions to have a mean of 0 and a variance of 1.

[0124] It is understood that this embodiment introduces action step information of a specific action category as structured prior knowledge, uses a bidirectional cross-attention mechanism to align the text features of the predicted action steps with the video features of the video, and then uses convolutional layer operations to obtain fused features, thereby enhancing the ability to capture subtle differences in actions.

[0125] The beneficial effect of this setup is that by setting up a first cross-attention module, a second cross-attention module, and a convolution module, the fusion features can be determined, ensuring the alignment of video to text and text to video, thereby improving the accuracy of the multimodal action evaluation model obtained through subsequent training.

[0126] S206. Input the fused features into the third sub-module to obtain the output prediction thought chain explanation.

[0127] In one possible implementation, the third submodule can be the architecture of the BERT model and include a decoder. For example, the self-attention mechanism of the BERT model in the third submodule has a causal attention mask to support autoregressive generation functionality.

[0128] For example, the third submodule generates the predictive thought chain interpretation token by token in an autoregressive manner. The autoregressive generation process stops when the decoder of the third submodule generates a predefined termination token ([EOS]) or reaches the set maximum generation length (e.g., 500 characters). Further, all generated token sequences are detoxified to obtain the final predictive thought chain interpretation.

[0129] Preferably, at time step t, the third submodule calculates the probability distribution of the next token and samples it based on all previously generated token sequences and fusion features. Its generation process may include:

[0130] (1) The third submodule receives all previously generated token sequences. As input, the query vector for the current step is computed through a self-attention layer with a causal mask. This process can be represented as:

[0131]

[0132] in, Represents word embedding layer, This represents a self-attention layer with a causal mask. This represents the output of the self-attention layer.

[0133] (2) Next, the third submodule uses in the cross-attention layer As a query, and using the fusion features as keys and values, a fusion process is performed. This process can be represented as:

[0134]

[0135] in, Indicates a cross-attention layer. This represents the output of the cross-attention layer. This indicates the fusion feature.

[0136] (3) Pass the final hidden state through a linear layer and normalize ( The function, mapped to the entire vocabulary space, yields the conditional probability that the next word in the current context is the i-th word in the vocabulary. This process can be represented as:

[0137]

[0138] in, express function, This represents the output of the cross-attention layer. This represents the weights in the normalization function of the third submodule. This represents the bias in the normalization function of the third submodule.

[0139] (4) Select a token from the probability distribution using a decoding strategy such as greedy search. This will be the output of the current time step. Then... Append to the end of the generated sequence to form a new sequence. Then, proceed to the next time step and repeat the above process.

[0140] It is understood that this embodiment introduces the concept of thought chain, and instead of directly generating simple and isolated action suggestions, it simulates the thought process of human experts to generate a complete reasoning path, reveal the complex causal relationships between steps, identify the root cause of errors, provide possible corrective explanations, and clarify the reasons behind action evaluation.

[0141] The beneficial effect of this setup is that, based on the autoregressive generation method of the third submodule, the output predictive thought chain explanation retains the temporal causal logic of the fused features, which aligns with the logic of humans to observe, analyze, and make suggestions in sequence, and also improves the reliability of the output predictive thought chain explanation.

[0142] S207. Based on the predicted action classification, predicted action quality, predicted thought chain interpretation, label information, and thought chain interpretation information, train the initial model to obtain the trained multimodal action evaluation model.

[0143] In one alternative implementation, step S207 may include:

[0144] The first loss value is determined based on the predicted action classification and the action classification label in the label information; the second loss value is determined based on the predicted action quality and the action quality label in the label information; the third loss value is determined based on the predicted thought chain interpretation and the thought chain interpretation information; the comprehensive loss value is determined based on the first loss value, the second loss value, and the third loss value; the initial model is trained based on the comprehensive loss value to obtain the trained multimodal action evaluation model.

[0145] It is understandable that the initial model training simultaneously learns three tasks: action classification evaluation, action quality evaluation, and thought chain interpretation evaluation, in order to improve the accuracy of the multimodal action evaluation model obtained through training.

[0146] For example, for the task of action classification and evaluation, a cross-entropy loss function can be used for supervision. This loss function guides the model to learn how to classify actions in a video. The first loss value can be expressed as:

[0147] ;

[0148] In the above formula, This represents the output of the cross-entropy loss function, i.e., the first loss value, where M represents the total number of action categories. This represents a symbolic function, which is 1 if the true class of the sample is c, and 0 otherwise. The probability that the i-th sample belongs to class c is predicted by the initial model.

[0149] For example, for the task of motion quality assessment, a binary cross-entropy loss function can be used for supervision. This loss function enables the initial model to assess the standardization of motion. The second loss value can be expressed as:

[0150] ;

[0151] In the above formula, This represents the output of the binary cross-entropy loss function, i.e., the second loss value. The label represents the actual action quality of the sample (e.g., 1 represents non-standard, 0 represents standard). This represents the probability that the action quality of the sample predicted by the initial model is non-standard.

[0152] For example, for the task of evaluating thought chain explanations, a language modeling loss function with label smoothing can be used for supervision. This loss function supervises the initial model to generate thought chain explanations word by word in an autoregressive manner, while label smoothing improves the generalization ability and fluency of the generated thought chain explanations. The third loss value can be expressed as:

[0153] ;

[0154] In the above formula, This represents the output of the language modeling loss function, i.e., the third loss value. Indicates the label smoothing factor. This represents the actual text token of the sample at time step t. This represents all real text tokens of the sample before time step t. Indicates fusion characteristics, This refers to the size of the vocabulary list.

[0155] Furthermore, by weighted summing the loss values ​​from the three tasks, we can obtain the comprehensive loss value. The comprehensive loss value can be expressed as:

[0156] ;

[0157] In the above formula, This represents the overall loss value. This represents the first loss value. This represents the second loss value. This represents the third loss value. The weight hyperparameter represents the preset contribution of the balance loss (for example, it can be 3).

[0158] For example, if the overall loss value meets the preset conditions, it indicates that the trained multimodal action evaluation model has been obtained; otherwise, the step of training the initial model based on the video and corresponding label information and thought chain explanation information is repeated until the overall loss value meets the preset conditions. For example, the preset conditions may be that the overall loss value is less than a preset loss threshold (e.g., 0.35).

[0159] The beneficial effect of this setup is that by combining the loss functions of the three tasks and making full use of the semantic consistency constraints between the generation of action classification, action quality and thought chain interpretation, the initial model can promote each other and complement each other's errors during the optimization process, thereby improving the accuracy and generalization performance of the multimodal action evaluation model obtained through training.

[0160] The training method for the multimodal motion evaluation model provided in this application considers applying thought chain interpretation information to the model training, helping the model to better analyze and deduce the motion quality in videos, thereby improving the accuracy of the multimodal motion evaluation model in motion quality assessment. The method of this application aims to improve the accuracy of motion quality assessment.

[0161] Figure 3 This is a schematic diagram of the architecture of the training method for the multimodal action evaluation model provided in this application. Based on the foregoing embodiments, this embodiment provides a detailed description of the logical architecture of the training method for the multimodal action evaluation model, such as... Figure 3 As shown, the architecture includes a multimodal fusion feature module and a multi-task prediction head, wherein the multi-task prediction head includes two MLPs and a text generation prediction head.

[0162] In this process, a video is used as input, and uniform sampling is performed to obtain a frame sequence. Based on this frame sequence, video features can be obtained through processing by a target video encoder (also known as a visual encoder). Simultaneously, the video is input into an MLP (also known as a visual large language model) responsible for action classification prediction to obtain a predicted action classification. Then, based on this predicted action classification, the corresponding action step information (also known as standard technical steps) is determined from the action step information set. This is the predicted action step, which can then be processed by a target text encoder (also known as a text encoder) to obtain text features.

[0163] By inputting video features and text features into the multimodal feature module of the initial model, fused features can be obtained.

[0164] The multimodal feature module can include two layers of cross-attention modules (also known as the first cross-attention module and the second cross-attention module). The cross-attention modules are connected to the residual network respectively. The outputs of the two residual networks are concatenated and then processed by the convolutional network for convolution and normalization to obtain the fused features.

[0165] Furthermore, based on the fusion features and the multi-task prediction head, the predicted action classification, predicted action quality, and predicted thought chain interpretation output by the multi-task prediction head can be obtained. Among them, the two MLPs in the multi-task prediction head (also known as the first sub-module and the second sub-module) realize action classification prediction and action quality prediction, respectively, while the text generation prediction head (also known as the third sub-module) realizes thought chain interpretation prediction (also known as thought chain-based text synthesis).

[0166] In the text generation prediction head, the fused features are taken as input and can be processed sequentially through causal mask self-attention, residual network, layer normalization and feedforward neural network to obtain the output prediction thought chain interpretation.

[0167] The architecture provided in this embodiment can execute the methods provided in the above method embodiments. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0168] Figure 4 A schematic diagram of the structure of the training device for the multimodal motion evaluation model provided in this application is shown below. Figure 4 As shown, the training device 40 for the multimodal motion evaluation model provided in this embodiment includes: a first processing unit 401, a second processing unit 402, and a training unit 403.

[0169] The first processing unit 401 is used to acquire a video dataset and determine an action step information set based on the video dataset. The video dataset includes videos and corresponding label information. The label information includes action classification labels and action quality labels. The action step information set includes action classification labels and corresponding action step information. The action step information represents the action technical steps under the action category.

[0170] The second processing unit 402 is used to determine the thought chain explanation information corresponding to each video in the video dataset based on the action step information set; wherein, the thought chain explanation information represents the text description and analysis suggestions of the action phenomenon in the video;

[0171] Training unit 403 is used to train the initial model based on the video and the corresponding label information and thought chain explanation information to obtain the trained multimodal action evaluation model; wherein, the multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

[0172] In an optional example, the first processing unit 401 is further configured to determine the action name corresponding to each action classification label in the video dataset;

[0173] Input the preset first prompt and action name into the preset first large model to obtain the output action step information; wherein, the preset first large model is used to output action step information;

[0174] Each action category label and its corresponding action step information are defined as an action step information set.

[0175] In an optional example, the second processing unit 402 is further configured to determine the action step information corresponding to the video from the action step information set based on the action classification label corresponding to the video.

[0176] The preset second prompt, video, and corresponding action step information and action quality labels are input into the preset second large model to obtain the thought chain explanation information corresponding to the output video; the preset second large model is used to output the thought chain explanation information.

[0177] In an optional example, the initial model includes a first submodule, a second submodule, and a third submodule; the training unit 403 is also used to train the initial model based on the video and corresponding label information and thought chain explanation information to obtain a trained multimodal action evaluation model, including:

[0178] Determine the video features of the video; where video features characterize the semantic representation of the video in the spatiotemporal dimension;

[0179] The video features are input into the first submodule to obtain the output predicted action classification, and the video features are input into the second submodule to obtain the output predicted action quality. Here, the predicted action classification represents the prediction result of the action classification information of the video, and the predicted action quality represents the confidence level of whether the action is standard or not.

[0180] Based on video features and predicted action classification, fusion features are determined; whereby, fusion features characterize the fusion result of video semantics and action step semantics;

[0181] The fused features are input into the third submodule to obtain the output prediction thought chain interpretation;

[0182] The initial model is trained based on the predicted action classification, predicted action quality, predicted thought chain interpretation, label information, and thought chain interpretation information to obtain a trained multimodal action evaluation model.

[0183] In an optional example, training unit 403 is further configured to determine the predicted action steps corresponding to the video from the action step information set based on the predicted action classification; wherein the predicted action steps represent the action step information corresponding to the predicted action classification.

[0184] Based on the predicted action steps, text features are determined; whereby text features represent the semantic vector of the predicted action steps.

[0185] The video features and text features are fused together to obtain the fused features.

[0186] In an optional example, the initial model includes a first cross-attention module, a second cross-attention module, and a convolutional module; the training unit 403 is also used to determine text enhancement features based on the first cross-attention module, using text features as queries and video features as keys and values.

[0187] Using video features as queries and text features as keys and values, the video enhancement features are determined based on the second cross-attention module;

[0188] Based on text enhancement features and video enhancement features, fusion features are determined using convolutional modules.

[0189] In an optional example, training unit 403 is also used to determine a first loss value based on the action classification label in the predicted action classification and label information;

[0190] The second loss value is determined based on the predicted action quality and the action quality label in the label information;

[0191] The third loss value is determined based on the predicted thought chain explanation and the thought chain explanation information;

[0192] The comprehensive loss value is determined based on the first loss value, the second loss value, and the third loss value.

[0193] Based on the comprehensive loss value, the initial model is trained to obtain a trained multimodal action evaluation model.

[0194] The training device for the multimodal motion evaluation model provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0195] Figure 5 A schematic diagram of the structure of the electronic device provided in this application, such as... Figure 5 As shown, the electronic device 50 provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device 50 further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0196] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0197] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0198] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0199] The memory may include high-speed memory (Random Access Memory, RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0200] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0201] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0202] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0203] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0204] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in application-specific integrated circuits (ASICs). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0205] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0206] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0208] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0209] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0210] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A training method for a multimodal action evaluation model, characterized in that, include: Obtain a video dataset and determine an action step information set based on the video dataset; wherein, the video dataset includes videos and corresponding tag information, the tag information includes action classification tags and action quality tags, the action step information set includes action classification tags and corresponding action step information, and the action step information represents the action technical steps under the action category; For each video in the video dataset, the thought chain explanation information corresponding to the video is determined based on the action step information set; wherein, the thought chain explanation information represents the textual description and analysis suggestions of the action phenomenon in the video; Based on the video and its corresponding tag information and thought chain explanation information, the initial model is trained to obtain a trained multimodal action evaluation model; wherein, the multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

2. The method according to claim 1, characterized in that, Based on the video dataset, a set of action step information is determined, including: For each action classification label in the video dataset, determine the action name corresponding to the action classification label; The preset first prompt and the action name are input into the preset first large model to obtain the output action step information; wherein, the preset first large model is used to output the action step information; Each action category label and its corresponding action step information are determined as the action step information set.

3. The method according to claim 1, characterized in that, Based on the action step information set, determine the thought chain explanation information corresponding to the video, including: Based on the action classification tags corresponding to the video, determine the action step information corresponding to the video from the action step information set; The preset second prompt, the video, the corresponding action step information, and the action quality label are input into the preset second large model to obtain the output thought chain explanation information corresponding to the video; wherein, the preset second large model is used to output the thought chain explanation information.

4. The method according to any one of claims 1-3, characterized in that, The initial model includes a first submodule, a second submodule, and a third submodule. Based on the video and corresponding tag information and thought chain explanation information, the initial model is trained to obtain a trained multimodal action evaluation model, including: Determine the video features of the video; wherein the video features characterize the semantic representation of the video in the spatiotemporal dimension; The video features are input into the first submodule to obtain the output predicted action classification, and the video features are input into the second submodule to obtain the output predicted action quality; wherein, the predicted action classification represents the prediction result of the action classification information of the video, and the predicted action quality represents the confidence level of whether the action is standard or not; Based on the video features and the predicted action classification, a fusion feature is determined; wherein the fusion feature represents the fusion result of video semantics and action step semantics; The fused features are input into the third submodule to obtain the output predictive thought chain explanation; The initial model is trained based on the predicted action classification, the predicted action quality, the predicted thought chain explanation, the label information, and the thought chain explanation information to obtain a trained multimodal action evaluation model.

5. The method according to claim 4, characterized in that, Based on the video features and the predicted action classification, the fusion features are determined, including: Based on the predicted action classification, the predicted action steps corresponding to the video are determined from the action step information set; wherein, the predicted action steps represent the action step information corresponding to the predicted action classification. Based on the predicted action steps, text features are determined; wherein the text features represent the semantic vector of the predicted action steps; The video features and the text features are fused together to obtain the fused features.

6. The method according to claim 5, characterized in that, The initial model includes a first cross-attention module, a second cross-attention module, and a convolutional module; the video features and the text features are fused to obtain the fused features, including: Using the text features as the query and the video features as the key and value, text enhancement features are determined based on the first cross-attention module; Using the video features as the query and the text features as the key and value, video enhancement features are determined based on the second cross-attention module; Based on the text enhancement features and the video enhancement features, the fusion features are determined using the convolutional module.

7. The method according to claim 4, characterized in that, Based on the predicted action classification, the predicted action quality, the predicted thought chain interpretation, the label information, and the thought chain interpretation information, the initial model is trained to obtain a trained multimodal action evaluation model, including: A first loss value is determined based on the predicted action classification and the action classification label in the label information; A second loss value is determined based on the predicted action quality and the action quality label in the label information; Based on the predicted thought chain explanation and the thought chain explanation information, a third loss value is determined; A comprehensive loss value is determined based on the first loss value, the second loss value, and the third loss value. Based on the comprehensive loss value, the initial model is trained to obtain a trained multimodal action evaluation model.

8. A training device for a multimodal motion evaluation model, characterized in that, include: The first processing unit is used to acquire a video dataset and determine an action step information set based on the video dataset; wherein the video dataset includes videos and corresponding tag information, the tag information includes action classification tags and action quality tags, the action step information set includes action classification tags and corresponding action step information, and the action step information represents the action technical steps under the action category; The second processing unit is used to determine the thought chain explanation information corresponding to each video in the video dataset based on the action step information set; wherein the thought chain explanation information represents the text description and analysis suggestions of the action phenomenon in the video; The training unit is used to train the initial model based on the video and the corresponding tag information and thought chain explanation information to obtain the trained multimodal action evaluation model; wherein, the multimodal action evaluation model is used to evaluate the classification and quality of actions in the video and provide interpretability analysis.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images