Multi-modal action recognition method and system based on fine-grained analysis and knowledge distillation

Abstract

The present application relates to the field of computer vision, deep learning and multi-modal behavior analysis, and particularly relates to a multi-modal action recognition method and system based on fine-grained analysis and knowledge distillation, which comprises: frame extraction on an input video to obtain an image sequence; body part segmentation on the images and inputting the CLIP model combined with text description to obtain multi-modal features of each part; integrating feature information through a part-level fusion network; constructing a student model combined with a learnable prompt and performing knowledge distillation; optimizing and training the model and outputting an action recognition result. The present application well integrates fine-grained visual features and semantic information, effectively mines key patterns in actions, and thus improves the accuracy of action recognition.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and artificial intelligence, and particularly relates to a motion recognition method and system using fine-grained feature analysis, multi-modal information fusion, and a teacher-student knowledge distillation framework. BACKGROUND

[0002] With the rapid progress of artificial intelligence and computer vision technology, motion recognition as a key technology is gradually becoming the focus of research and application. In today's highly digitalized era, the amount of video data is showing an explosive growth trend, covering multiple fields from social media, security monitoring to medical diagnosis. The vast amount of video data contains rich human behavior information, and accurately recognizing and understanding these behaviors is crucial for many application scenarios. For example, in an intelligent security system, quickly and accurately detecting abnormal behavior can effectively prevent criminal incidents; in the medical care field, accurately capturing patient movements can assist doctors in making more accurate diagnoses and rehabilitation effect evaluations; and in human-computer interaction scenarios, understanding user body language can make machine-human communication more natural and smooth, improving user experience. However, human behavior itself has high complexity and diversity, coupled with various uncertain factors in the actual environment, which makes it a great challenge to achieve high-precision motion recognition. Traditional computer vision methods based on rules or feature extraction often struggle to meet the actual demand when faced with complex actions that are similar in appearance but different in meaning. Fortunately, with the development of deep learning algorithms, especially visual-linguistic pre-training models like CLIP, new hope has been brought to solve this problem. Such advanced tools not only can understand and interpret the semantic meaning behind image content more deeply, but also can perform well in handling subtle changes in dynamic scenarios. Therefore, with the help of these cutting-edge technical means for motion recognition, not only can the recognition accuracy be greatly improved, but also the efficiency and experience in multiple industry fields can be promoted, thus opening up a new era full of unlimited possibilities.

[0003] Currently, one of the main challenges in the field of action recognition is how to effectively capture the subtle differences in complex actions. Many actions are very similar in appearance, especially in dynamic scenes, and this similarity is even more pronounced. Take the two classic weightlifting actions in sports competitions, "snatch" and "clean and jerk", as examples. Although they both end up with the barbell being held over the head, there are significant differences in the specific execution process, especially in the movement of the arms. These differences are reflected in the specific steps of the action, the power points of different parts of the body, and the timing rhythm. For "snatch", the athlete needs to quickly lift the barbell from the ground to shoulder height, then quickly complete the turnover and lock the position; while "clean and jerk" requires the athlete to first place the barbell in front of the chest, and then use the explosive power of the legs to lift it to the air. Therefore, accurately distinguishing between these two actions is not only crucial for the technical analysis of athletes, but also puts higher requirements on the accuracy of action recognition algorithms. Unfortunately, most existing action recognition methods tend to use global image analysis, which often ignores the subtle changes in different parts of the body, especially the arms, when performing specific actions. Such coarse-grained analysis may make it difficult for the model to accurately capture those subtle but critical action features, leading to confusion in classification. SUMMARY

[0004] To overcome the above existing technical problems, the present application proposes a multi-modal action recognition method based on fine-grained analysis and knowledge distillation. The steps of the method include:

[0005] S1: receiving an action video as the original input data, performing frame extraction processing on the input video to obtain a sequence of key frame images, and taking the extracted image sequence as the input data for subsequent processing;

[0006] S2: the teacher model first performs body part segmentation on the input image, inputs the segmented image and the automatically generated corresponding text description into the CLIP model, and obtains the image features, text features and class scores of each body part;

[0007] S3: the teacher model then integrates the features and scores of each body part obtained in S2 through a part-level fusion network to obtain the final teacher image features, teacher text features and teacher class scores;

[0008] S4: the student model connects the input image with the learnable visual prompt, connects the automatically generated text description with the learnable text prompt, and inputs the CLIP model to obtain the student image features, student text features and student class scores;

[0009] S5: Aligning the teacher features and student features and the teacher class scores and student class scores using knowledge distillation, so that the student model learns the knowledge representation and classification ability of the teacher model;

[0010] S6: Jointly optimizing the student model in combination with the knowledge distillation loss and the classification task loss;

[0011] S7: Using the optimized student model to classify the input action video, calculating the action class probability distribution, and determining and outputting the most likely action class as the recognition result according to the calculated probability distribution.

[0012] Further, in step S1, the input video is frame-extracted by a fixed-interval dynamic frame-extraction strategy to obtain a key frame image sequence.

[0013] Further, the specific implementation of step S2 includes:

[0014] S21: Obtain the bounding box coordinates of each main body part of the input image according to the dataset annotation, the main body parts including head, left and right arms, left and right hands, torso, left and right legs, and left and right feet, crop the original image according to these bounding box coordinates to obtain ten body part sub-images; adjust the size of each sub-image and keep three color channels;

[0015] S22: Automatically generate a corresponding text description based on the input image, which summarizes the action content in the image, and the generated text description follows the following template: {person's [body part name] is in [action name] [part action]};

[0016] S23: Input the preprocessed body part sub-images and the generated text description into the pre-trained CLIP model respectively to obtain the image feature vpart, the text feature tpart and the class score logitspart of each body part.

[0017] Further, the CLIP model includes two core components: an image encoder and a text encoder. The image encoder adopts an improved Vision Transformer (ViT) architecture to extract high-level visual feature representations. The text encoder is a Transformer-based text processing model for generating rich text feature representations.

[0018] Further, the specific implementation of step S3 includes:

[0019] S31: input the image features vpart of each body part obtained in step S2 into an MLP network for fusion, which learns the relationship between different body part features and generates a comprehensive teacher image feature vteacher;

[0020] S32: input the text features tpart of each body part obtained in step S2 into another MLP network for processing, and finally obtain a comprehensive teacher text feature tteacher;

[0021] S33: for the class score, a weighted average method is used for fusion, which assigns a weight to the class score logitspart of each body part, and obtains the final teacher class score logitsteacher.

[0022] Further, the specific implementation of step S4 includes:

[0023] S41: splice the input original image with the pre-designed learnable visual prompt, and the spliced image contains the original information and additional visual clues, to obtain an enhanced image;

[0024] S42: based on the input image, generate a corresponding concise text description, following the template: a photo of {action}; then, connect the generated text with the learnable text prompt to obtain an enhanced text;

[0025] S43: input the enhanced image and text into the CLIP model sharing the weights with the teacher model, the CLIP model uses its image encoder to process the enhanced image, uses the text encoder to process the enhanced text, to obtain the student image feature vstudent, the student text feature tstudent, and the student class score logitsstudent.

[0026] Further, the knowledge distillation in step S5 includes:

[0027] S51: knowledge transfer is achieved by minimizing the L1 loss between the teacher image feature vteacher and the student image feature vstudent;

[0028] Lossimage=‖vstudent-vteacher‖

[0029] S52: similar to the image feature distillation, the knowledge transfer of the text feature is also achieved by minimizing the L1 loss;

[0030] Losstext=‖tstudent-tteacher‖

[0031] S53: Category score distillation: For the knowledge transfer of category scores, the KL divergence is adopted as the measurement standard, the KL divergence between the category score distribution logitsteacher output by the teacher model and the category score distribution logitsstudent output by the student model is calculated, and it is minimized;

[0032] Losslogits=KL(logitsteacher‖logitsstudent)

[0033] S54: Combine the above three distillation losses: image feature L1 loss, text feature L1 loss and category score KL divergence to form a comprehensive knowledge distillation loss function Loss K nowledge D istillation, and different weights are assigned to each loss term to balance the importance of learning in different aspects:

[0034] Loss K nowledge D istillation=αLossimage+βLosstext+γLosslogits

[0035] Wherein, α, β, γ are weights, Lossimage, Losstext, Losslogits represent image distillation loss, text distillation loss and category score distillation loss respectively.

[0036] Further, the specific implementation of step S6 includes:

[0037] S61: Cross-entropy loss is used as the training loss of the classification task to optimize the classification ability of the student model. The loss function first calculates the cosine similarity between the image feature v and the text feature t Then calculate the cross-entropy loss based on the similarity;

[0038]

[0039] Wherein, N represents the batch size, τ is the temperature hyperparameter, vi and ti represent the image feature and text feature of the i-th sample respectively;

[0040] S62: Joint optimization of knowledge distillation loss and classification loss of S61:

[0041] Losstotal=Loss K nowledge D istillation+Loss CE

[0042] wherein, Loss K nowledge D istillation is a knowledge distillation loss function;

[0043] S63: using a stochastic gradient descent optimizer to optimize the parameters of the student model.

[0044] The application also provides a multi-modal action recognition system based on fine-grained analysis and knowledge distillation, comprising:

[0045] one or more processors;

[0046] a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement a multi-modal action recognition method based on fine-grained analysis and knowledge distillation as described in the above technical solutions.

[0047] Compared with the prior art, the multi-modal action recognition method based on fine-grained analysis and knowledge distillation provided by the application effectively improves the accuracy and robustness of action recognition by in-depth analysis of fine-grained information of body parts. This method combines intra-modal information and inter-modal interaction to achieve efficient fusion of multi-modal data. In addition, using the knowledge distillation technology, the application can fully learn fine-grained features in the training stage, and only needs standard input in the test stage, significantly reducing the dependence on complex labeled data. This design not only improves the applicability of the model in real scenarios, but also enhances the recognition ability in data-scarce environments, making our action recognition method perform well in diversified applications and have stronger practical value. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is the overall flowchart of the application;

[0049] Figure 2 is the CLIP structure diagram adopted by the application.

[0050] Figure 3 is the multi-modal action recognition method based on fine-grained analysis and knowledge distillation of the application. DETAILED DESCRIPTION

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings, and the described embodiments are only some of the embodiments of the application, not all. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the application.

[0052] The embodiment of the application provides a multi-modal action recognition method based on fine-grained analysis and knowledge distillation. Figure 1 As shown in the figure, the method steps include:

[0053] S1: receiving an action video as original input data, performing frame extraction processing on the input video to obtain a key frame image sequence, and taking the extracted image sequence as input data for subsequent processing.

[0054] In step S1, the application receives an action video as original input data and performs frame extraction processing to obtain a key frame image sequence. By using a fixed interval dynamic frame extraction strategy, representative key frames are selected from the entire video to reduce the computational burden and improve the efficiency of subsequent processing. These extracted key frame image sequences will be used as input data for subsequent steps to ensure that the model can effectively capture and recognize the key details of the action and provide high-quality basic data for subsequent body part segmentation and feature extraction.

[0055] S2: the teacher model first performs body part segmentation on the input image, inputs the segmented image and the automatically generated corresponding text description into the CLIP model, and obtains the image features, text features and category scores of each body part.

[0056] In step S2, to achieve fine-grained analysis of action recognition, first, according to the annotation information of the data set, the boundary box coordinates of each main body part in the input image are obtained. These body parts include the head, left and right arms, left and right hands, torso, left and right legs, and left and right feet. Then, the original image is cropped using these boundary box coordinates to obtain ten body part sub-images. Each cropped sub-image is adjusted to 224x224 pixels to ensure that it meets the input requirements of the CLIP model while maintaining three color channels for subsequent processing.

[0057] Step S21, to achieve fine-grained action recognition, first, according to the annotation information of the data set, the boundary box coordinates of each main body part in the input image are obtained. These body parts include the head, left and right arms, left and right hands, torso, left and right legs, and left and right feet. Through the labeled data, the model can accurately locate the position of each body part in the image. Next, the original image is cropped according to these boundary box coordinates to obtain ten corresponding body part sub-images. Each sub-image will be adjusted to 224x224 pixels and maintain three color channels, which can ensure that it meets the input requirements of the CLIP model.

[0058] Step S22, based on the input image, we automatically generate corresponding text descriptions. These text descriptions summarize the action content in the image, following a specific template: "{name of [body part] of [person] in [action name] [part action]}." For example, for an action of a person jumping, the text description might be "the head of the person moves upward in jumping." This structured description provides necessary contextual information for subsequent feature extraction, allowing the model to better understand the meaning of the action when processing the image.

[0059] Step S23 involves inputting the pre-processed body part sub-image and the generated text description into a pre-trained CLIP model, respectively. In this process, the image encoder and text encoder of the CLIP model play a key role. The image encoder is usually based on an improved Vision Transformer (ViT) architecture, which divides the input image into multiple small blocks (patches) and performs linear embedding on each small block. Through multiple layers of self-attention mechanisms, it learns the global and local features of the image, and extracts high-level visual feature representations. At the same time, the text encoder is based on the Transformer structure, which can process the semantic information of the input text and generate rich text feature representations. The Transformer captures the context relationship in the text through the self-attention mechanism, ensuring the integrity and accuracy of the text information. Through the collaborative work of image and text encoders, the CLIP model can map image and text information to a shared feature space, allowing the model to perform cross-modal understanding and matching. Finally, this process will generate image features vpart, text features tpart and class scores logitspart for each body part, laying a solid foundation for subsequent action recognition.

[0060] Vision Transformer (ViT) is a model architecture that treats image processing as a sequential task. It first divides the input image into fixed-size small blocks (patches), then flattens and embeds these small blocks into a high-dimensional space. By adding position encoding, ViT can preserve spatial information. Next, multiple self-attention layers and feedforward neural networks are stacked together, allowing the model to capture the relationship between global and local features in the image, and finally generate feature representations for classification. ViT has successfully surpassed traditional convolutional neural networks in various visual tasks with its strong performance and flexibility.

[0061] Transformer is a model based on self-attention mechanism, widely used in natural language processing. It converts input text into vector embeddings and adds position encoding to maintain word order information. The core component is the self-attention mechanism, which allows the model to dynamically adjust the representation according to the relationship between words in the input sequence. Multiple self-attention heads work in parallel, allowing the model to capture different context information simultaneously. Transformer uses multiple layers of stacking combined with feedforward neural networks to form a deep structure, greatly improving the model's expression ability, and is widely used in machine translation, text generation and sentiment analysis tasks.

[0062] S3: The teacher model then integrates the features and scores of each body part obtained in S2 through a part-level fusion network to obtain the final teacher image features, teacher text features and teacher class scores.

[0063] In the S3 step, the features and scores of each body part obtained from S2 are integrated through a part-level fusion network. The purpose of this process is to effectively combine the information of each body part to form more representative teacher image features, teacher text features and teacher class scores. Through this integration, the model can capture the mutual relationship between each body part and improve the understanding of the whole action. The final teacher features not only contain fine-grained information, but also enhance the accuracy of action classification.

[0064] In the S31 step, the image features vpart obtained for each body part are input into a multi-layer perceptron (MLP) network for fusion. MLP is composed of multiple fully connected layers, and the neurons of each layer are connected to all the neurons of the adjacent layer. This structure allows MLP to fully learn the complex relationships between different body part features, thereby generating a comprehensive teacher image feature vteacher. By using a nonlinear activation function, MLP enhances the model's expression ability, making the feature fusion process more flexible and effective. This feature fusion strategy ensures that the final generated image features can more comprehensively reflect the details of the entire action.

[0065] In the S32 step, the text features tpart of each body part obtained in S2 are input into another MLP network for processing. The main role of this network is to integrate the semantics of the text descriptions related to different parts, ultimately generating a comprehensive teacher text feature tteacher. By inputting the text features into the MLP, the model can effectively capture the deep semantic relationship between each body part and its corresponding action. This process not only improves the understanding of text information, but also enables the final text features to better combine with image features, thereby supporting more accurate action recognition.

[0066] In step S33, the teacher model fuses the class scores logitspart using a weighted average method. Each body part's class score is assigned a corresponding weight according to its importance, resulting in a comprehensive teacher class score logitsteacher. This weighting method allows the model to more accurately reflect the contribution of each body part in the final classification. In this way, the teacher model can more effectively integrate information from different parts when processing action recognition tasks, improving the accuracy and reliability of classification. Overall, the fusion of class scores further enhances the model's understanding of complex actions, making the final recognition result more accurate.

[0067] S4: Connect the input image with the learnable visual cue, and connect the automatically generated text description with the learnable text cue, input the CLIP model to obtain the student image feature, student text feature and student class score.

[0068] In step S4, the student model connects the input image with the learnable visual cue and the automatically generated text description with the learnable text cue, and then inputs them into the CLIP model to obtain the student image feature, student text feature and student class score. The core of this process is to enhance the model's performance in action recognition tasks through cueing.

[0069] In step S41, the original input image is spliced with the learnable visual cue. The purpose of this visual cue is to guide the model to focus on key areas or features in the image, thereby improving its understanding of actions. Through splicing, the generated image contains both original information and additional visual clues, which helps the model highlight important parts related to actions during feature extraction. The learning ability of the visual cue allows the model to dynamically adjust its focus according to the training data, thereby achieving higher recognition accuracy when dealing with complex actions.

[0070] In step S42, a concise text description corresponding to the input image is generated, following the template: "a photo of {action}." This description summarizes the main content of the image, making the text information more direct and clear. Then, the generated text is connected with the learnable text cue to enhance the semantic expression of the text. The learnable text cue can provide additional context information for the model, helping it better understand and process action descriptions. This text enhancement method not only improves the quality of text features, but also further optimizes the model's performance in understanding actions.

[0071] In step S43, the enhanced images and texts are input into the CLIP model that shares weights with the teacher model. The image encoder of the CLIP model processes these enhanced images to extract richer student image features vstudent, while the text encoder processes the enhanced texts to generate corresponding student text features tstudent. In this process, the input has been enhanced with prompts, and the processing is similar to that of the teacher model, but the student model can generate different student class scores logitsstudent due to the newly introduced visual and textual prompts. This method ensures that the model can utilize prompt information during feature extraction to improve the overall performance of action recognition.

[0072] S5: Aligning features and class scores using knowledge distillation to enable the student model to learn the knowledge representation and classification ability of the teacher model.

[0073] In step S5, knowledge distillation is used to align the features and class scores between the teacher model and the student model, enabling the student model to effectively learn the knowledge representation and classification ability of the teacher model. This process ensures consistency in the features and classification results of the student model, thereby improving overall performance.

[0074] In step S51, knowledge transfer is achieved by minimizing the L1 loss between the teacher image features vteacher and the student image features vstudent. Specifically, the L1 distance between these two features is calculated and used as one of the optimization objectives. Such loss calculation encourages the student model to gradually learn similar image representation capabilities as the teacher model, helping to capture key visual information and ensuring that the student model can achieve comparable levels of image feature representation as the teacher model. The formula is:

[0075] Lossimage = ‖vstudent - vteacher‖

[0076] Similar to the distillation of image features, in step S52, the knowledge transfer of text features is also achieved by minimizing the L1 loss. Here, the L1 distance between the teacher text features tteacher and the student text features tstudent is calculated and included in the overall optimization objective. This process enables the student model to better understand and represent text information, improving its cross-modal understanding ability and maintaining consistency with the teacher model in text processing. The formula is:

[0077] Losstext = ‖tstudent - tteacher‖

[0078] In step S53, for the knowledge transfer of class scores, the KL divergence is used as the measurement standard. By calculating the KL divergence between the class score distribution logitsstudent output by the student model and the class score distribution logitsteacher output by the teacher model, and minimizing it. This method not only transfers the final classification result, but also preserves the relative relationship information between different categories, which helps the student model to learn more detailed classification knowledge and ensures that the accuracy of class recognition reaches the level of the teacher model. The formula is:

[0079] Losslogits=KL(logitsteacher‖logitsstudent)

[0080] In step S54, the above three distillation losses (image feature L1 loss, text feature L1 loss and class score KL divergence) are combined to form a comprehensive knowledge distillation loss function. Different weights are assigned to each loss term to balance the importance of learning in different aspects. This comprehensive loss function ensures balanced learning in multiple dimensions and improves the effectiveness of knowledge transfer. The formula is:

[0081] Loss K nowledge D istillation=αLossimage+βLosstext+γLosslogits

[0082] L1 loss reflects the distance between two feature representations by measuring the average of absolute differences between two vectors, which is suitable for feature-level knowledge transfer. KL divergence focuses on the difference between two probability distributions, which is particularly suitable for class score knowledge transfer. By combining these two measurement methods, the knowledge distillation process can effectively transfer the knowledge of the teacher model to the student model.

[0083] S6: Combine the knowledge distillation loss and the classification task loss to jointly optimize the learnable prompts in the student model.

[0084] In step S6, the student model is jointly optimized by combining the knowledge distillation loss and the classification task loss, aiming to improve its knowledge transfer ability and classification performance.

[0085] In step S61, the cross-entropy loss (CE Loss) is used as the training loss of the classification task, aiming to optimize the classification ability of the student model. The loss function first calculates the cosine similarity between the image feature v and the text feature t, and the formula is:

[0086]

[0087] After calculating the similarity, the cross-entropy loss is obtained by the following formula:

[0088]

[0089] Where N represents the batch size, τ is the temperature hyperparameter, and vi and ti represent the image feature and text feature of the i-th sample, respectively. By introducing the temperature parameter, the model can adjust the smoothness of the distribution, thereby improving the robustness of classification.

[0090] In step S62, the knowledge distillation loss Loss K nowledge D istillation and the classification loss Loss CE are combined to form the total loss function, as follows:

[0091] Losstotal=Loss K nowledge D istillation+Loss CE

[0092] This joint optimization strategy ensures that the student model learns the knowledge representation of the teacher model while achieving better performance on the classification task, ultimately improving the generalization ability of the entire model.

[0093] In step S63, the stochastic gradient descent (SGD) optimizer is used to optimize the model parameters. The initial learning rate is set to 0.0025, and the entire training process is performed for 50 rounds, with each round performing a complete traversal of the entire training data set. After calculating the gradient on each batch of data, the model parameters are updated, and the model performance is gradually improved through iterative optimization. The SGD optimizer adjusts the learning rate and momentum and other hyperparameters to enable the model to quickly converge in a complex feature space, improving training efficiency and effectiveness. This optimization strategy enables the student model to quickly adapt and improve classification accuracy when dealing with new data.

[0094] S7: Use the optimized student model to classify the input action video and calculate the action category probability distribution. According to the calculated probability distribution, determine and output the most likely action category as the recognition result.

[0095] In step S7, the optimized student model is used to classify the input action video, and the most likely action category is determined and output as the final result.

[0096] On the other hand, the embodiments of the present application also provide a multi-modal action recognition system based on fine-grained analysis and knowledge distillation, comprising:

[0097] one or more processors;

[0098] a memory device storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement a multi-modal action recognition method based on fine-grained analysis and knowledge distillation as described in the above technical solutions.

[0099] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by programs instructing relevant hardware, and the programs can be stored in a computer-readable storage medium, which can include ROM, RAM, magnetic disks or optical disks, etc.

[0100] As shown in Table 1, we evaluated the performance of the model on Kinetics-TPS, which is a subset of 3,809 videos from the Kinetics-700 dataset, containing detailed multi-dimensional annotations, including 10 body parts, 74 part states and 75 interactive objects, covering 24 categories of human actions, and using four different training-test ratios (2:8, 4:6, 6:4, 8:2) and four input video frame numbers (4, 8, 16, 32) to explore the performance of the model under different sample sizes and video information density. The experimental results show that the method achieves the best performance on this dataset. This process not only has important significance in the technical field, but also plays a positive role in social applications. Through efficient action recognition technology, it can be widely used in many fields, such as intelligent monitoring, security protection, motion analysis and human-computer interaction, etc. In the field of public security, accurate identification of suspicious behavior or abnormal action can help to timely warn and improve the level of social security protection. In the field of sports analysis, coaches and athletes can optimize training effectiveness and improve competitive level through detailed action feedback. In addition, this technology can also be used in smart home, medical health and other scenarios to monitor the daily activities of the elderly or patients and provide real-time feedback to help them maintain their ability to live independently.

[0101] Table 1 Results achieved by the present application on Kinetic-TPS dataset

[0102]

[0103] The above examples further illustrate the purpose, technical solutions and advantages of the present application. It should be understood that the above examples are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made to the present application within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal action recognition method based on fine-grained analysis and knowledge distillation, characterized in that, Includes the following steps: S1: Receive motion video as raw input data, perform frame extraction processing on the input video to obtain key frame image sequence, and use the extracted image sequence as input data for subsequent processing. S2: The teacher model first segments the input image into body parts, and then inputs the segmented images and automatically generated corresponding text descriptions into the CLIP model to obtain the image features, text features and category scores for each body part. The specific implementation of step S2 includes: S21: Obtain the bounding box coordinates of each major body part in the input image based on the dataset annotations. The major body parts include the head, left and right arms, left and right forearms, torso, left and right legs, and left and right feet. Crop the original image based on these bounding box coordinates to obtain these ten body part sub-images. Resize each sub-image while maintaining 3 color channels. S22: Automatically generate a corresponding text description based on the input image. This text description summarizes the action content in the image. The generated text description follows the template: {[body part name] of the person in [action name] [body part action]}; S23: Input the preprocessed body part sub-images and generated text descriptions into the pre-trained CLIP model to obtain the image features of each body part. Text features and category scores ; S3: The teacher model then integrates the features and scores of each body part obtained in S2 through a part-level fusion network to obtain the final teacher image features, teacher text features, and teacher category score. S4: The student model connects the input image with learnable visual cues and the automatically generated text description with learnable text cues. The input is then fed into the CLIP model to obtain student image features, student text features, and student category scores. S5: Use knowledge distillation to align teacher features with student features and teacher category scores with student category scores, enabling the student model to learn the knowledge representation and classification ability of the teacher model; S6: Combine knowledge distillation loss and classification task loss to jointly optimize the student model; S7: Use the optimized student model to classify the input action video, calculate the probability distribution of action categories, and determine and output the most likely action category as the recognition result based on the calculated probability distribution.

2. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: In step S1, the input video is processed by a fixed-interval dynamic frame extraction strategy to obtain a keyframe image sequence.

3. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The CLIP model comprises two core components: an image encoder and a text encoder. The image encoder employs an improved Vision Transformer (ViT) architecture to extract high-level visual feature representations. The text encoder is a Transformer-based text processing model used to generate rich text feature representations.

4. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S3 includes: S31: Analyze the image features of each body part obtained in step S2. The input is fused into an MLP network, which learns the relationships between features of different body parts and generates a comprehensive teacher image feature set. ; S32: Obtain the text features of each body part obtained in step S2. It is also input into another MLP network for processing, ultimately yielding a comprehensive set of teacher text features. ; S33: For category scores, a weighted average method is used for fusion, which involves calculating the category scores for each body part. Assign a weight to obtain the final teacher category score. .

5. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S4 includes: S41: The original input image is stitched together with pre-designed learnable visual cues. The stitched image contains the original information and additional visual cues, resulting in an enhanced image. S42: Based on the input image, generate a corresponding concise text description, following the template: a photo of {action}; then, connect this generated text with learnable text prompts to obtain the enhanced text; S43: The enhanced image and text are input into the CLIP model, which shares weights with the teacher model. The CLIP model uses its image encoder to process the enhanced image and its text encoder to process the enhanced text, thus obtaining the student image features. Student text characteristics Student category score .

6. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The knowledge distillation in step S5 includes: S51: By minimizing teacher image features Student image features Knowledge transfer is achieved through L1 loss between the two; ； S52: Similar to image feature distillation, knowledge transfer of text features is also achieved by minimizing the L1 loss; ； S53: Category Score Distillation: For knowledge transfer of category scores, KL divergence is used as a metric to calculate the distribution of category scores output by the teacher model. Distribution of category scores output by the student model Minimize the KL divergence between them; ； S54: Combine the three distillation losses mentioned above—image feature L1 loss, text feature L1 loss, and category score KL divergence—to form a comprehensive knowledge distillation loss function. Furthermore, different weights are assigned to each loss term to balance the importance of different learning aspects: ； in, , , As weight, , , These represent image distillation loss, text distillation loss, and category score distillation loss, respectively.

7. The multimodal action recognition method based on fine-grained analysis and knowledge distillation according to claim 1, characterized in that: The specific implementation of step S6 includes: S61: Cross-entropy loss is used as the training loss for the classification task to optimize the classification ability of the student model. This loss function first calculates the image features. and text features Cosine similarity between Then, based on this similarity, the cross-entropy loss is calculated; ； in, Indicates batch size, For temperature hyperparameters, and Representing the first Image and text features of each sample; S62: Combine the knowledge distillation loss and the classification loss of S61 for joint optimization: ； in, The knowledge distillation loss function; S63: Use a stochastic gradient descent optimizer to optimize the parameters of the student model.

8. A multimodal action recognition system based on fine-grained analysis and knowledge distillation, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a multimodal action recognition method based on fine-grained analysis and knowledge distillation as described in any one of claims 1 to 7.

Images