Multi-modal action recognition method and system based on fine-grained analysis and knowledge distillation

Abstract

The invention relates to the field of computer vision, deep learning and multi-modal behavior analysis, in particular to a multi-modal action recognition method and system based on fine-grained analysis and knowledge distillation, and the method comprises the steps: carrying out the frame extraction of an input video, and obtaining an image sequence; carrying out body part segmentation on the image and inputting the image into a CLIP model in combination with text description to obtain multi-modal features of each part; integrating the feature information through a part level fusion network; a student model is constructed in combination with learnable prompts, and knowledge distillation is carried out; and carrying out optimization training on the model and outputting an action recognition result. According to the method, fine-grained visual features and semantic information are well fused, and key modes in actions are effectively mined, so that the accuracy of action recognition is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence, and in particular relates to an action recognition method and system using fine-grained feature analysis, multimodal information fusion and a teacher-student knowledge distillation framework. Background Art

[0002] With the rapid advancement of artificial intelligence and computer vision technology, action recognition as a key technology is gradually becoming a focus of research and application. In today's highly digital era, the amount of video data is showing an explosive growth trend, covering a variety of fields from social media, security monitoring to medical diagnosis. The massive amount of video data contains rich information about human behavior, and accurately identifying and understanding these behaviors is crucial for many application scenarios. For example, in intelligent security systems, rapid and accurate detection of abnormal behavior can effectively prevent the occurrence of criminal incidents; in the field of healthcare, accurate capture of patients' movements can assist doctors in making more accurate diagnoses and evaluating rehabilitation effects; and in the scenario of human-computer interaction, understanding the user's body language can make the communication between machines and people more natural and smooth, improving the user experience. However, human behavior itself is extremely complex and diverse, coupled with various uncertainties in the actual environment, which makes it a huge challenge to achieve high-precision action recognition. Traditional computer vision methods based on rules or feature extraction often seem powerless when faced with complex actions that look similar but have different meanings, and it is difficult to meet the standards of actual needs. Fortunately, with the development of deep learning algorithms, especially visual-language pre-training models like CLIP, new hope has been brought to solving this problem. Such advanced tools can not only understand and interpret the semantic meaning behind the image content at a deeper level, but also perform well in handling subtle changes in dynamic scenes. Therefore, using these cutting-edge technical means to perform action recognition can not only greatly improve recognition accuracy, but also promote efficiency improvement and experience optimization in multiple industries, thus opening a new era full of infinite possibilities.

[0003] Currently, a major challenge in the field of action recognition is how to effectively capture the subtle differences in complex actions. Many actions are very similar in appearance, especially in dynamic scenes. Take the two classic weightlifting actions of "snatch" and "clean and jerk" in sports events as an example. Although they both lift the barbell over the head in the final effect, there are significant differences in the specific execution process, especially in the performance of the arm movements. These differences are reflected in the specific steps of the action, the force points of different parts of the body, and the timing. For the "snatch", the athlete needs to quickly pull the barbell from the ground to shoulder height, then quickly flip and lock the position; while the "clean and jerk" requires the athlete to first place the barbell in front of the chest and then push it into the air through the explosive force of the legs. Therefore, accurately distinguishing between these two actions is not only crucial for the technical analysis of athletes, but also puts higher requirements on improving the accuracy of action recognition algorithms. Unfortunately, most existing action recognition methods tend to use global image analysis, which often ignores the subtle changes in various parts of the body (especially the arms) when performing specific actions. Such coarse-grained analysis may make it difficult for the model to accurately capture those subtle but critical action features, thus causing confusion in classification. Summary of the invention

[0004] In order to overcome the above existing problems, the present invention proposes a multimodal action recognition method based on fine-grained analysis and knowledge distillation, the method comprising the following steps:

[0005] S1: receiving an action video as raw input data, extracting frames from the input video to obtain a key frame image sequence, and using the extracted image sequence as input data for subsequent processing;

[0006] S2: The teacher model first segments the input image into body parts, and then inputs the segmented image and the automatically generated corresponding text description into the CLIP model to obtain the image features, text features, and category scores of each body part;

[0007] S3: The teacher model then integrates the features and scores of each body part obtained in S2 through the part-level fusion network to obtain the final teacher image features, teacher text features and teacher category scores;

[0008] S4: The student model connects the input image with the learnable visual cues, connects the automatically generated text description with the learnable text cues, and inputs into the CLIP model to obtain the student image features, student text features, and student category scores;

[0009] S5: Use knowledge distillation to align teacher features with student features and teacher category scores with student category scores, so that the student model can learn the knowledge representation and classification capabilities of the teacher model;

[0010] S6: Combine knowledge distillation loss and classification task loss to jointly optimize the student model;

[0011] S7: Use the optimized student model to classify the input action video, calculate the probability distribution of the action category, and determine and output the most likely action category as the recognition result based on the calculated probability distribution.

[0012] Furthermore, in step S1, a frame extraction process is performed on the input video through a dynamic frame extraction strategy at fixed intervals to obtain a key frame image sequence.

[0013] Furthermore, the specific implementation of step S2 includes:

[0014] S21: Obtain the bounding box coordinates of each major body part of the input image according to the dataset annotation, the major body parts include the head, left and right arms, left and right arms, torso, left and right legs, and left and right feet, and crop the original image according to these bounding box coordinates to obtain these ten body part sub-images; resize each sub-image and keep three color channels;

[0015] S22: Automatically generate a corresponding text description based on the input image, where the text description summarizes the action content in the image. The generated text description follows the following template: {[body part name] of the person in [action name] [body part action]};

[0016] S23: The preprocessed body part sub-image and the generated text description are respectively input into the pre-trained CLIP model to obtain the image feature vpart, text feature tpart and category score logitspart of each body part.

[0017] Furthermore, the CLIP model includes two core components: an image encoder and a text encoder. The image encoder adopts an improved Vision Transformer (ViT) architecture to extract high-level visual feature representations; the text encoder is a Transformer-based text processing model for generating rich text feature representations.

[0018] Furthermore, the specific implementation of step S3 includes:

[0019] S31: Input the image features vpart of each body part obtained in step S2 into the MLP network for fusion. The MLP network learns the relationship between the features of different body parts and generates a comprehensive teacher image feature vteacher;

[0020] S32: The text features tpart of each body part obtained in step S2 are also input into another MLP network for processing, and finally a comprehensive teacher text feature tteacher is obtained;

[0021] S33: For the category scores, the weighted average method is used for fusion. This process assigns a weight to the category score logitspart of each body part to obtain the final teacher category score logitsteacher.

[0022] Furthermore, the specific implementation of step S4 includes:

[0023] S41: splicing the input original image with the pre-designed learnable visual cues, the spliced ​​image contains the original information and additional visual cues, and the enhanced image is obtained;

[0024] S42: Based on the input image, generate a corresponding concise text description, following the template: a photo of {action}; then, concatenate this generated text with the learnable text prompt to obtain the enhanced text;

[0025] S43: The enhanced image and text are respectively input into the CLIP model that shares weights with the teacher model. The CLIP model uses its image encoder to process the enhanced image and uses the text encoder to process the enhanced text to obtain the student image features vstudent, the student text features tstudent, and the student category score logitsstudent.

[0026] Furthermore, the knowledge distillation in step S5 includes:

[0027] S51: Knowledge transfer is achieved by minimizing the L1 loss between the teacher image feature vteacher and the student image feature vstudent;

[0028] Lossimage=‖vstudent-vteacher‖

[0029] S52: Similar to image feature distillation, knowledge transfer of text features is also achieved by minimizing L1 loss;

[0030] Losstext=‖tstudent-teacher‖

[0031] S53: Category score distillation: For the knowledge transfer of category scores, KL divergence is used as the metric to calculate the KL divergence between the category score distribution logitsteacher output by the teacher model and the category score distribution logitsstudent output by the student model and minimize it;

[0032] Losslogits=KL(logitsteacher‖logitsstudent)

[0033] S54: Combine the above three distillation losses: image feature L1 loss, text feature L1 loss and category score KL divergence to form a comprehensive knowledge distillation loss function Loss K nowledge D istillation, and assigns different weights to each loss term to balance the importance of learning in different aspects:

[0034] Loss K nowledge D istillation=αLossimage+βLosstext+γLosslogits

[0035] Among them, α, β, and γ are weights, Lossimage, Losstext, and Losslogits represent image distillation loss, text distillation loss, and category score distillation loss, respectively.

[0036] Furthermore, the specific implementation of step S6 includes:

[0037] S61: Cross entropy loss is used as the training loss for the classification task to optimize the classification ability of the student model. The loss function first calculates the cosine similarity between the image feature v and the text feature t. Then the cross entropy loss is calculated based on this similarity;

[0038]

[0039] Where N represents the batch size, τ is the temperature hyperparameter, vi and ti represent the image features and text features of the i-th sample respectively;

[0040] S62: Combine knowledge distillation loss and classification loss of S61 for joint optimization:

[0041] Losstotal=Loss K nowledge D istillation+loss CE

[0042] Among them, Loss K nowledge D istillation is the knowledge distillation loss function;

[0043] S63: Use stochastic gradient descent optimizer to optimize the parameters of the student model.

[0044] The present invention also provides a multimodal action recognition system based on fine-grained analysis and knowledge distillation, comprising:

[0045] one or more processors;

[0046] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a multimodal action recognition method based on fine-grained analysis and knowledge distillation as described in the above technical solution.

[0047] Compared with the prior art, the multimodal action recognition method based on fine-grained analysis and knowledge distillation proposed in the present invention effectively improves the accuracy and robustness of action recognition by deeply analyzing the fine-grained information of body parts. This method combines the internal information of the modality and the interaction between the modalities to achieve efficient fusion of multimodal data. In addition, by utilizing the knowledge distillation technology, the present invention can fully learn fine-grained features in the training phase, and only requires standard input in the testing phase, significantly reducing the dependence on complex labeled data. This design not only improves the applicability of the model in real scenarios, but also enhances the recognition ability in data-scarce environments, making our action recognition method perform well in a variety of applications and have stronger practical value. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the overall flow chart of the present invention;

[0049] Figure 2 This is a diagram of the CLIP structure used in the present invention.

[0050] Figure 3 This is a multimodal action recognition method based on fine-grained analysis and knowledge distillation of the present invention. DETAILED DESCRIPTION

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] The embodiment of the present invention provides a multimodal action recognition method based on fine-grained analysis and knowledge distillation, such as Figure 1 As shown, the method steps include:

[0053] S1: receiving an action video as original input data, extracting frames from the input video to obtain a key frame image sequence, and using the extracted image sequence as input data for subsequent processing.

[0054] In step S1, the present invention receives the action video as the original input data and performs frame extraction processing on it to obtain a key frame image sequence. Through a dynamic frame extraction strategy at fixed intervals, representative key frames are selected from the entire video to reduce the computational burden and improve the efficiency of subsequent processing. These extracted key frame image sequences will be used as input data for subsequent steps to ensure that the model can effectively capture and identify the key details of the action, providing high-quality basic data for subsequent body part segmentation and feature extraction.

[0055] S2: The teacher model first segments the input image into body parts, and inputs the segmented image and the corresponding automatically generated text description into the CLIP model to obtain the image features, text features and category scores of each body part.

[0056] In step S2, in order to achieve action recognition with fine-grained analysis, the bounding box coordinates of each major body part in the input image are first obtained based on the annotation information of the dataset. These body parts include the head, left and right arms, left and right hands, torso, left and right legs, and left and right feet. Subsequently, the original image is cropped using these bounding box coordinates to obtain sub-images of ten body parts. Each cropped sub-image is resized to 224×224 pixels to ensure that it meets the input requirements of the CLIP model while maintaining three color channels for subsequent processing.

[0057] Step S21, in order to achieve fine-grained action recognition, it is first necessary to obtain the bounding box coordinates of each major body part in the input image based on the annotation information of the data set. These body parts include: head, left and right arms, left and right hands, torso, left and right legs, and left and right feet. By annotating the data, the model can accurately locate the position of each body part in the image. Next, the original image is cropped according to these bounding box coordinates to obtain ten corresponding body part sub-images. Each sub-image will be adjusted to 224×224 pixels and maintain three color channels to ensure that it matches the input requirements of the CLIP model.

[0058] In step S22, based on the input image, we automatically generate corresponding text descriptions. These text descriptions summarize the action content in the image and follow a specific template: "{person's [body part name] in [action name] [part action]}." For example, for an action of a person jumping, the text description may be "the person's head moves upward in the jump." This structured description provides the necessary contextual information for subsequent feature extraction, allowing the model to better understand the meaning of the action when processing the image.

[0059] Step S23 involves inputting the preprocessed body part sub-image and the generated text description into the pre-trained CLIP model respectively. In this process, the image encoder and text encoder of the CLIP model play a key role. The image encoder is usually based on the improved Vision Transformer (ViT) architecture, which divides the input image into multiple patches and linearly embeds each patch. It learns the global and local features of the image through a multi-layer self-attention mechanism to extract high-level visual feature representations. At the same time, the text encoder is based on the Transformer structure and can process the semantic information of the input text and generate rich text feature representations. Transformer captures the contextual relationship in the text through the self-attention mechanism to ensure the integrity and accuracy of the text information. Through the collaborative work of the image and text encoders, the CLIP model can map the image and text information to a shared feature space, so that the model can understand and match across modalities. Ultimately, this process will generate image features vpart, text features tpart and category scores logitspart for each body part, laying a solid foundation for subsequent action recognition.

[0060] Vision Transformer (ViT) is a model architecture that treats image processing as a sequence task. It first divides the input image into small patches of fixed size, and then flattens and embeds these patches into a high-dimensional space. By adding position encoding, ViT is able to preserve spatial information. Next, multiple self-attention layers and feedforward neural networks are stacked together, allowing the model to capture the relationship between global and local features in the image and finally generate feature representations for classification. With its powerful performance and flexibility, ViT has successfully surpassed traditional convolutional neural networks in various visual tasks.

[0061] Transformer is a model based on the self-attention mechanism and is widely used in natural language processing. It works by converting the input text into a vector embedding and adding positional encoding to maintain information about the order of words. The core component is the self-attention mechanism, which enables the model to dynamically adjust the representation based on the relationship between words in the input sequence. Multiple self-attention heads work in parallel, allowing the model to capture different contextual information at the same time. Transformer forms a deep structure by stacking multiple layers and combining it with a feedforward neural network, which greatly improves the expressiveness of the model and is widely used in tasks such as machine translation, text generation, and sentiment analysis.

[0062] S3: The teacher model then integrates the features and scores of each body part obtained in S2 through a part-level fusion network to obtain the final teacher image features, teacher text features, and teacher category scores.

[0063] In step S3, the features and scores of each body part obtained from S2 are integrated through a part-level fusion network. The purpose of this process is to effectively merge the information of each body part to form more representative teacher image features, teacher text features, and teacher category scores. Through this integration, the model is able to capture the relationship between various body parts and improve the understanding of the entire action. The final teacher features not only contain fine-grained information, but also enhance the accuracy of action classification.

[0064] In step S31, the image features vpart obtained from each body part are input into a multi-layer perceptron (MLP) network for fusion. MLP consists of multiple fully connected layers, and the neurons in each layer are all connected to the neurons in the adjacent layers. This structure allows MLP to fully learn the complex relationship between the features of different body parts, thereby generating a comprehensive teacher image feature vteacher. By using nonlinear activation functions, MLP enhances the expressive power of the model, making the feature fusion process more flexible and effective. This feature fusion strategy ensures that the final generated image features can more comprehensively reflect the details of the entire action.

[0065] In step S32, the text features tpart of each body part obtained in S2 are input into another MLP network for processing. The main function of this network is to semantically integrate the text descriptions related to different parts and finally generate a comprehensive teacher text feature tteacher. By inputting the text features into the MLP, the model can effectively capture the deep semantic relationship between each body part and its corresponding action. This process not only improves the ability to understand text information, but also enables the final text features to be better combined with image features, thereby supporting more accurate action recognition.

[0066] In step S33, the teacher model uses the weighted average method to fuse the category scores logitspart. The category score of each body part is assigned a corresponding weight according to its importance, and finally a comprehensive teacher category score logitsteacher is obtained. This weighted method enables the model to more accurately reflect the contribution of each body part in the final classification. In this way, the teacher model can more effectively integrate information from different parts when processing action recognition tasks, thereby improving the accuracy and reliability of classification. Overall, the fusion of category scores further enhances the model's ability to understand complex actions, making the final recognition results more accurate.

[0067] S4: Connect the input image with the learnable visual cues, connect the automatically generated text description with the learnable text cues, and input into the CLIP model to obtain the student image features, student text features and student category scores.

[0068] In step S4, the student model connects the input image with the learnable visual prompt and the automatically generated text description with the learnable text prompt, and then inputs it into the CLIP model to obtain the student image features, student text features and student category scores. The core of this process is to improve the performance of the model in the action recognition task through prompt enhancement.

[0069] In step S41, the original input image is concatenated with a learnable visual cue. This visual cue is designed to guide the model to focus on key areas or features in the image, thereby improving its ability to understand the action. Through concatenation, the generated image contains both the original information and additional visual cues, which helps the model to highlight important parts related to the action when extracting features. The learning ability of visual cues enables the model to dynamically adjust its focus based on the training data, thereby achieving higher recognition accuracy when processing complex actions.

[0070] In step S42, a corresponding concise text description is generated based on the input image, following the template: "A photo of {action}." This description summarizes the main content of the image, making the text information more direct and clear. Then, this generated text is connected with the learnable text prompt to enhance the semantic expression of the text. The learnable text prompt can provide additional contextual information for the model, helping the model to better understand and process the action description. This text enhancement method not only improves the quality of text features, but also further optimizes the performance of the model in understanding actions.

[0071] In step S43, the enhanced images and text are respectively input into the CLIP model that shares weights with the teacher model. The image encoder of the CLIP model processes these enhanced images to extract richer student image features vstudent, while the text encoder processes the enhanced text to generate corresponding student text features tstudent. In this process, since the input has been enhanced with prompts, it is similar to the processing of the teacher model, but due to the newly introduced visual and textual prompts, the student model is able to generate different student category scores logitsstudnet. This approach ensures that the model can utilize prompt information during feature extraction and improve the overall performance of action recognition.

[0072] S5: Use knowledge distillation to align features and category scores so that the student model can learn the knowledge representation and classification capabilities of the teacher model.

[0073] In step S5, knowledge distillation technology is used to align the features and category scores between the teacher model and the student model, so that the student model can effectively learn the knowledge representation and classification capabilities of the teacher model. This process ensures the consistency of the student model in features and classification results, thereby improving the overall performance.

[0074] In step S51, knowledge transfer is achieved by minimizing the L1 loss between the teacher image feature vteacher and the student image feature vstudent. Specifically, the L1 distance between the two features is calculated and used as one of the optimization targets. Such loss calculation prompts the student model to gradually learn image representation capabilities similar to the teacher model, which helps to capture key visual information and ensure that the student model can reach a level comparable to the teacher model in the representation of image features. The formula is:

[0075] Lossimage=‖vstudent-vteacher‖

[0076] Similar to the distillation of image features, in step S52, the knowledge transfer of text features is also achieved by minimizing the L1 loss. Here, the L1 distance between the teacher text feature tteacher and the student text feature tstudent is calculated and incorporated into the overall optimization goal. This process enables the student model to better understand and represent text information, improve its cross-modal understanding ability, and thus be consistent with the teacher model in text processing. The formula is:

[0077] Losstext=‖tstudent-teacher‖

[0078] In step S53, KL divergence is used as a metric for knowledge transfer of category scores. The KL divergence between the category score distribution logitsteacher output by the teacher model and the category score distribution logitsstudent output by the student model is calculated and minimized. This method not only transmits the final classification result, but also retains the relative relationship information between different categories, which helps the student model learn more detailed classification knowledge and ensures that the accuracy of category recognition reaches the level of the teacher model. The formula is:

[0079] Losslogits=KL(logitsteacher‖logitsstudent)

[0080] In step S54, the above three distillation losses (image feature L1 loss, text feature L1 loss and category score KL divergence) are combined to form a comprehensive knowledge distillation loss function. Different weights are assigned to each loss term to balance the importance of learning in different aspects. This comprehensive loss function ensures balanced learning of the model in multiple dimensions and improves the effect of knowledge transfer. The formula is:

[0081] Loss K nowledge D istillation=αLossimage+βLosstext+γLosslogits

[0082] L1 loss intuitively reflects the distance between two feature representations by measuring the average of the absolute difference between two vectors, which is suitable for feature-level knowledge transfer. KL divergence focuses on the difference between two probability distributions, which is particularly suitable for knowledge transfer of category scores. By combining these two measurement methods, the knowledge distillation process can fully and effectively transfer the knowledge of the teacher model to the student model.

[0083] S6: Combine knowledge distillation loss and classification task loss to jointly optimize the learnable hints in the student model.

[0084] In step S6, the student model is jointly optimized by combining knowledge distillation loss and classification task loss, aiming to simultaneously improve its knowledge transfer ability and classification performance.

[0085] In step S61, cross entropy loss (CE Loss) is used as the training loss for the classification task, aiming to optimize the classification ability of the student model. The loss function first calculates the cosine similarity between the image feature v and the text feature t, and the formula is:

[0086]

[0087] After calculating the similarity, the cross entropy loss is given by the following formula:

[0088]

[0089] Among them, N represents the batch size, τ is the temperature hyperparameter, and vi and ti represent the image features and text features of the i-th sample respectively. By introducing the temperature parameter, the model can adjust the smoothness of the distribution, thereby improving the robustness of the classification.

[0090] In step S62, the knowledge distillation loss Loss K nowledge D Istillation and classification loss CE Combine them to form the total loss function, the formula is as follows:

[0091] Losstotal=Loss K nowledge D istillation+loss CE

[0092] This joint optimization strategy ensures that the student model can achieve better performance on classification tasks while learning the knowledge representation of the teacher model, ultimately improving the generalization ability of the entire model.

[0093] In step S63, the stochastic gradient descent (SGD) optimizer is used to optimize the model parameters. The initial learning rate is set to 0.0025, and a total of 50 rounds are performed during the entire training process, with each round performing a complete traversal of the entire training data set. The model parameters are updated after calculating the gradient on each batch of data, and the model performance is gradually improved through this iterative optimization method. The SGD optimizer adjusts hyperparameters such as learning rate and momentum to enable the model to converge quickly in a complex feature space, thereby improving training efficiency and effectiveness. This optimization strategy enables the student model to quickly adapt and improve classification accuracy when dealing with new data.

[0094] S7: Use the optimized student model to classify the input action video, calculate the probability distribution of the action category, and determine and output the most likely action category as the recognition result based on the calculated probability distribution.

[0095] In step S7, the input action video is classified using the optimized student model, and the most likely action category is finally determined and output.

[0096] On the other hand, an embodiment of the present invention further provides a multimodal action recognition system based on fine-grained analysis and knowledge distillation, including:

[0097] one or more processors;

[0098] A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement a multimodal action recognition method based on fine-grained analysis and knowledge distillation as described in the above technical solution.

[0099] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, which may include: ROM, RAM, disk or CD, etc.

[0100] As shown in Table 1, we evaluated the performance of the model on Kinetics-TPS, which is a dataset selected from 3,809 videos in the Kinetics-700 dataset. It contains detailed multi-dimensional annotations, including 10 body parts, 74 body states, and 75 interactive objects, covering 24 types of human actions. Four different training-test ratios (2:8, 4:6, 6:4, 8:2) and four input video frame numbers (4, 8, 16, 32) are used to explore the performance of the model under different sample sizes and video information densities. The experimental results show that the method achieves the current best performance on this dataset. This process is not only of great significance at the technical level, but also plays a positive role in social applications. Through efficient action recognition technology, it can be widely used in many fields, such as intelligent monitoring, security prevention, motion analysis, and human-computer interaction. In the field of public safety, accurate identification of suspicious behaviors or abnormal actions helps to provide timely warnings and improve the level of social security protection. In motion analysis, coaches and athletes can optimize training effects and improve their competitive level through detailed action feedback. In addition, this technology can also monitor the daily activities of the elderly or patients in scenarios such as smart homes and medical health, provide real-time feedback, and help them maintain the ability to live independently.

[0101] Table 1 Results obtained by the present invention on the Kinetic-TPS dataset

[0102]

[0103] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A multimodal action recognition method based on fine-grained analysis and knowledge distillation, characterized in that: The following steps are involved: S1: receiving an action video as raw input data, extracting frames from the input video to obtain a key frame image sequence, and using the extracted image sequence as input data for subsequent processing; S2: The teacher model first segments the input image into body parts, and then inputs the segmented image and the automatically generated corresponding text description into the CLIP model to obtain the image features, text features, and category scores of each body part; S3: The teacher model then integrates the features and scores of each body part obtained in S2 through the part-level fusion network to obtain the final teacher image features, teacher text features and teacher category scores; S4: The student model connects the input image with the learnable visual cues, connects the automatically generated text description with the learnable text cues, and inputs into the CLIP model to obtain the student image features, student text features, and student category scores; S5: Use knowledge distillation to align teacher features with student features and teacher category scores with student category scores, so that the student model can learn the knowledge representation and classification capabilities of the teacher model; S6: Combine knowledge distillation loss and classification task loss to jointly optimize the student model; S7: Use the optimized student model to classify the input action video, calculate the probability distribution of the action category, and determine and output the most likely action category as the recognition result based on the calculated probability distribution.

2. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: In step S1, a frame extraction process is performed on the input video through a dynamic frame extraction strategy at fixed intervals to obtain a key frame image sequence.

3. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S2 includes: S21: Obtain the bounding box coordinates of each major body part of the input image according to the dataset annotation, the major body parts include the head, left and right arms, left and right arms, torso, left and right legs, and left and right feet, and crop the original image according to these bounding box coordinates to obtain these ten body part sub-images; resize each sub-image and keep three color channels; S22: Automatically generate a corresponding text description based on the input image, where the text description summarizes the action content in the image. The generated text description follows the following template: {[body part name] of the person in [action name] [body part action]}; S23: The preprocessed body part sub-image and the generated text description are respectively input into the pre-trained CLIP model to obtain the image feature vpart, text feature tpart and category score logitspart of each body part.

4. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 3 is characterized in that: The CLIP model includes two core components: an image encoder and a text encoder. The image encoder uses an improved Vision Transformer (ViT) architecture to extract high-level visual feature representations. The text encoder is a Transformer-based text processing model used to generate rich text feature representations.

5. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S3 includes: S31: Input the image features vpart of each body part obtained in step S2 into the MLP network for fusion. The MLP network learns the relationship between the features of different body parts and generates a comprehensive teacher image feature vteacher; S32: The text features tpart of each body part obtained in step S2 are also input into another MLP network for processing, and finally a comprehensive teacher text feature tteacher is obtained; S33: For the category scores, the weighted average method is used for fusion. This process assigns a weight to the category score logitspart of each body part to obtain the final teacher category score logitsteacher.

6. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S4 includes: S41: splicing the input original image with the pre-designed learnable visual cues, the spliced ​​image contains the original information and additional visual cues, and the enhanced image is obtained; S42: Based on the input image, generate a corresponding concise text description, following the template: a photo of {action}; then, concatenate this generated text with the learnable text prompt to obtain the enhanced text; S43: The enhanced image and text are respectively input into the CLIP model that shares weights with the teacher model. The CLIP model uses its image encoder to process the enhanced image and uses the text encoder to process the enhanced text to obtain the student image features vstudent, the student text features tstudent, and the student category score logitsstudent.

7. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The knowledge distillation in step S5 includes: S51: Knowledge transfer is achieved by minimizing the L1 loss between the teacher image feature vteacher and the student image feature vstudent; Lossimage=‖vstudent-vteacher‖ S52: Similar to image feature distillation, knowledge transfer of text features is also achieved by minimizing L1 loss; Losstext=‖tstudent-teacher‖ S53: Category score distillation: For the knowledge transfer of category scores, KL divergence is used as the metric to calculate the KL divergence between the category score distribution logitsteacher output by the teacher model and the category score distribution logitsstudent output by the student model and minimize it; Losslogits=KL(logitsteacher‖logitsstudent) S54: Combine the above three distillation losses: image feature L1 loss, text feature L1 loss and category score KL divergence to form a comprehensive knowledge distillation loss function Loss K nowledge D istillation, and assigns different weights to each loss term to balance the importance of learning in different aspects: Loss K knowledge D distillation=αLossimage+βLosstext+γLosslogits Among them, α, β, and γ are weights, Lossimage, Losstext, and Losslogits represent image distillation loss, text distillation loss, and category score distillation loss, respectively.

8. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S6 includes: S61: Cross entropy loss is used as the training loss for the classification task to optimize the classification ability of the student model. The loss function first calculates the cosine similarity between the image feature v and the text feature t. Then the cross entropy loss is calculated based on this similarity; Where N represents the batch size, τ is the temperature hyperparameter, vi and ti represent the image features and text features of the i-th sample respectively; S62: Combine knowledge distillation loss and classification loss of S61 for joint optimization: Total loss=Loss K knowledge D distillation+Loss CE Among them, Loss K nowledge D istillation is the knowledge distillation loss function; S63: Use stochastic gradient descent optimizer to optimize the parameters of the student model.

9. A multimodal action recognition system based on fine-grained analysis and knowledge distillation, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement a multimodal action recognition method based on fine-grained analysis and knowledge distillation as described in any one of claims 1 to 8.

Images