Small sample motion recognition method, electronic device and medium

Abstract

The present application discloses a small sample action recognition method, electronic device and medium, the method comprising: obtaining a support set video, a query set video and a category label template; performing feature processing on the support set video, the query set video and the category label template through a preset multimodal training model to obtain a first visual feature vector, a class-level semantic feature and a second visual feature vector; inputting the above three data into a proxy semantic diffusion module for noise diffusion processing; performing feature representation on the first instance-level proxy semantics, the class-level semantic features and the first visual feature vector output by the proxy semantic diffusion module to construct a category prototype, and performing feature fusion on the second instance-level proxy semantics to obtain a query feature representation; performing action recognition on the query feature representation and the category prototype to determine the action type of the second action video frame in the query set video. The embodiments of the present invention can capture instance-level semantic details and improve action recognition accuracy.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a small sample action recognition method, electronic equipment and medium. Background Art

[0002] Small-shot action recognition aims to learn and recognize human actions using only a small number of annotated video clips, which is itself a video understanding task under data-scarce conditions. Traditional small-shot action recognition methods usually rely on unimodal pre-trained models. Unimodal methods usually rely only on RGB video data for feature extraction. These methods mainly rely on the visual modality for transfer learning, while ignoring the multimodal association between vision and language, and often have low generalization to unseen action categories.

[0003] In recent years, the rise of large-scale multimodal pre-training models, especially the application of the Contrastive Language-Image Pre-training (CLIP) model, has brought new possibilities for small-sample action recognition. Multimodal pre-training models combine the semantic information of vision and language to provide richer semantic understanding. However, the problem with existing multimodal methods is that they still rely mainly on unified semantics at the class level, while ignoring the diversity of semantic information at the instance level.

[0004] Therefore, the limitation of current methods in feature modeling lies in their over-reliance on visual modalities or class-level semantics, failing to fully utilize the rich semantic information at the instance level, and in small-sample action recognition tasks, simply aligning instance-level video features with class-level action name text embeddings, failing to fully exploit the semantic diversity at the instance level, resulting in reduced action recognition accuracy. Summary of the invention

[0005] The embodiments of the present application provide a small sample action recognition method, an electronic device, and a storage medium, which can capture instance-level semantic details and improve action recognition accuracy.

[0006] In a first aspect, an embodiment of the present application provides a small sample action recognition method, including:

[0007] Obtain a support set video, a query set video, and a category label template, wherein the support set video includes multiple frames of first action video frames, the query set video includes multiple frames of second action video frames, and the category label template stores an action category name corresponding to the first action video frame;

[0008] The support set video, the query set video and the category label template are feature processed by a preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, wherein the multimodal training model includes a proxy semantic diffusion module, a feature fusion module and a prototype matching module;

[0009] Inputting the first visual feature vector, the class-level semantic feature, and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing, and outputting a first instance-level proxy semantic corresponding to the first visual feature vector and a second instance-level proxy semantic corresponding to the second visual feature vector;

[0010] Performing feature representation on the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector through the feature fusion module to construct a category prototype corresponding to the support set video, and performing feature fusion on the second instance-level proxy semantics through the feature fusion module to obtain a query feature representation;

[0011] The prototype matching module performs action recognition on the query feature representation and the category prototype to determine the action type of the second action video frame in the query set video.

[0012] In some embodiments, the multimodal training model includes a shared visual encoder and a text encoder; the feature processing of the support set video, the query set video, and the category label template by the preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature, and a second visual feature vector corresponding to each second action video frame includes:

[0013] Input the support set video into a preset multimodal training model, so that the shared visual encoder performs feature extraction on the first action video frame in the support set video, outputs a first visual feature vector corresponding to each first action video frame, and the text encoder performs feature extraction on the category label template, and outputs a class-level semantic feature;

[0014] The query set video is input into a preset multimodal training model so that the shared visual encoder extracts features from the second action video frames of the query set video and outputs a second visual feature vector corresponding to each second action video frame.

[0015] In some embodiments, the semantic diffusion module includes a first semantic diffusion submodule and a second semantic diffusion submodule; the step of inputting the first visual feature vector, the class-level semantic feature, and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing, and outputting a first instance-level proxy semantic corresponding to the first visual feature vector and a second instance-level proxy semantic corresponding to the second visual feature vector, comprises:

[0016] The second visual feature vector is gradually diffused by the second semantic diffusion submodule to obtain second instance-level proxy semantics;

[0017] Forward diffusion of the class-level semantic features is performed through the first semantic diffusion submodule to obtain noisy semantic features and embedded representations;

[0018] The first visual feature vector, the noisy semantic feature and the embedded representation are input into the first semantic diffusion submodule for reverse diffusion, and the first instance-level proxy semantics is output.

[0019] In some embodiments, the feature fusion module includes an intra-frame fusion layer and an inter-frame fusion layer; the feature fusion module performs feature representation on the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector to construct a category prototype corresponding to the support set video, including:

[0020] Inputting the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector into the intra-frame fusion layer for visual semantic feature fusion, and outputting frame information;

[0021] Inputting the frame information into the inter-frame fusion layer, so that the inter-frame fusion layer establishes a temporal relationship between different frames according to the frame information and outputs a feature representation;

[0022] A mean operation is performed on feature representations of the same category in the support set video to construct a category prototype corresponding to the support set video.

[0023] In some embodiments, the step of performing feature fusion on the second instance-level proxy semantics by the feature fusion module to obtain a query feature representation includes:

[0024] fusing the second instance-level proxy semantics frame by frame through the intra-frame fusion layer to obtain a fusion result;

[0025] The temporal relationship between different frames in the fusion result is established through the inter-frame fusion layer to obtain the query feature representation.

[0026] In some embodiments, performing action recognition on the query feature representation and the category prototype by the prototype matching module to determine the action type of the second action video frame in the query set video includes:

[0027] Inputting the query feature representation and the category prototype into the prototype matching module for similarity matching to obtain a matching result;

[0028] Perform action recognition on the second action video frame in the query set video according to the matching result to determine the action type of the second action video frame in the query set video.

[0029] In some embodiments, after inputting the first visual feature vector, the class-level semantic feature, and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing and outputting a first instance-level proxy semantic corresponding to the first visual feature vector and a second instance-level proxy semantic corresponding to the second visual feature vector, the method further includes:

[0030] The agent semantic diffusion module is trained by a preset supervised loss function.

[0031] In some embodiments, the noisy semantic feature is calculated by the following formula:

[0032] ;

[0033] in, It is k The noise factor of the step, It is a class-level semantic feature; N(0,1) is the noise that follows the standard normal distribution.

[0034] In some embodiments, the first instance-level proxy semantics is calculated by the following formula:

[0035] ;

[0036] in, is the intermediate instance-level proxy semantics of the k-th step and the t-th frame, is the first visual feature vector of the tth frame in the support set video, σ k is the noise standard deviation of the k-th step of the back diffusion process, Z N(0,I) is random noise, is a denoising neural network, It is k The noise factor of the step, is a time interpolator used to capture the temporal changes of the first instance-level proxy semantics.

[0037] In a second aspect, an embodiment of the present application provides an electronic device comprising at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the small sample action recognition method as described in the first aspect.

[0038] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the small sample action recognition method as described in the first aspect.

[0039] In some embodiments, the small sample action recognition method, electronic device and storage medium provided by the embodiments of the present application have at least the following beneficial effects: first, the support set video, query set video and category label template are obtained, and then the support set video, query set video and category label template are feature processed through a preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, so as to realize the extraction of video features of the support set video and the query set video, and at the same time realize the extraction of semantic features. After that, the first visual feature vector, the class-level semantic feature and the second visual feature vector are input into the proxy semantic diffusion module for noise diffusion processing, and the first instance-level proxy semantics corresponding to the first visual feature vector and the second instance-level proxy semantics corresponding to the second visual feature vector are output. Through a multi-step pre-training knowledge guidance mechanism and a time-conditional reverse diffusion step, instance-level temporal semantic information is gradually learned from the video data itself, the original video label semantics are expanded and enriched, and the capture of rich dynamic semantic information is realized, providing richer semantic expression for action recognition. The feature fusion module is used to represent the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector, so as to realize the combination of visual and semantic features, so as to construct a category prototype corresponding to the support set video, which is convenient for subsequent prototype matching, and the feature fusion module is used to perform feature fusion on the second instance-level proxy semantics to obtain the query feature representation. Since the category label in the query set video is unknown, the embodiment of the present application performs action recognition on the query feature representation and the category prototype through the prototype matching module, so as to realize the matching of the second visual feature vector of the query set video with the category prototype of the support set video, so as to determine the action type of the second action video frame in the query set video, and realize the accurate recognition of the action type. The embodiment of the present application increases the diversity of semantic representation in terms of the richness and temporality of the semantic content, and gradually generates instance-level proxy semantic features with temporal dynamics through a multi-step pre-training knowledge guidance mechanism and a time-conditional back diffusion step, so as to realize the temporal semantic representation of the action video instance level, which can better capture the fine-grained semantic information implicit in the video, and jointly model this information with the visual features, thereby improving the overall generalization ability and robustness of the small sample action recognition system. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 is a specific method flow chart of a small sample action recognition method provided by an embodiment of the present application;

[0041] Figure 2 It is a flowchart of a specific method for performing feature processing on support set videos, query set videos and category label templates through a preset multimodal training model provided by an embodiment of the present application;

[0042] Figure 3 This is a flowchart of a specific method for inputting a first visual feature vector, a class-level semantic feature, and a second visual feature vector into a proxy semantic diffusion module for noise diffusion processing provided by an embodiment of the present application;

[0043] Figure 4 It is a flowchart of a specific method for performing feature representation on the first instance-level proxy semantics, the class-level semantic features and the first visual feature vector through a feature fusion module provided by an embodiment of the present application;

[0044] Figure 5 It is a flowchart of a specific method for performing feature fusion on second instance-level proxy semantics through a feature fusion module provided in an embodiment of the present application;

[0045] Figure 6 This is a flowchart of a specific method for performing action recognition on query feature representation and category prototypes through a prototype matching module provided in an embodiment of the present application;

[0046] Figure 7 is a specific method flow chart of a small sample action recognition method provided by another embodiment of the present application;

[0047] Figure 8 It is a schematic diagram of the model architecture of a multimodal training model provided in an example of this application;

[0048] Fig. 9 is a schematic diagram of a reverse diffusion process provided by an example of the present application;

[0049] Fig.10 This is a performance comparison diagram of the ProLIPS framework provided in an example of this application under the SSv2-Full, Kinetics and UCF101 datasets;

[0050] Fig.11 This is a schematic diagram of the performance comparison of the ProLIPS framework provided in an example of this application under the SSv2-Small and HMDB51 datasets;

[0051] Fig.12 It is a schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] Small-shot action recognition aims to learn and recognize human actions using only a small number of annotated video clips, which is itself a video understanding task under data-scarce conditions. Traditional small-shot action recognition methods usually rely on unimodal pre-trained models. Unimodal methods usually rely only on RGB video data for feature extraction. These methods mainly rely on the visual modality for transfer learning, while ignoring the multimodal association between vision and language, and often have low generalization to unseen action categories.

[0054] In recent years, the rise of large-scale multimodal pre-training models, especially the application of the Contrastive Language-Image Pre-training (CLIP) model, has brought new possibilities for small-sample action recognition. Multimodal pre-training models combine the semantic information of vision and language to provide richer semantic understanding. However, the problem with existing multimodal methods is that they still rely mainly on unified semantics at the class level, while ignoring the diversity of semantic information at the instance level.

[0055] Therefore, the limitation of current methods in feature modeling lies in their over-reliance on visual modalities or class-level semantics, failing to fully utilize the rich semantic information at the instance level, and in small-sample action recognition tasks, simply aligning instance-level video features with class-level action name text embeddings, failing to fully exploit the semantic diversity at the instance level, resulting in reduced action recognition accuracy.

[0056] In order to solve the above problems, the present embodiment provides a small sample action recognition method, device and storage medium, which have at least the following beneficial effects: first, a support set video, a query set video and a category label template are obtained, and then feature processing is performed on the support set video, the query set video and the category label template through a preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, so as to extract video features of the support set video and the query set video, and at the same time extract semantic features. Afterwards, the first visual feature vector, the class-level semantic feature and the second visual feature vector are input into a proxy semantic diffusion module for noise diffusion processing, and a first instance-level proxy semantic corresponding to the first visual feature vector and a second instance-level proxy semantic corresponding to the second visual feature vector are output. Through a multi-step pre-training knowledge guidance mechanism and a time-conditional reverse diffusion step, instance-level temporal semantic information is gradually learned from the video data itself, the original video label semantics is expanded and enriched, and the capture of rich dynamic semantic information is achieved, providing richer semantic expression for action recognition. The feature fusion module is used to represent the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector, so as to realize the combination of visual and semantic features, so as to construct a category prototype corresponding to the support set video, which is convenient for subsequent prototype matching, and the feature fusion module is used to perform feature fusion on the second instance-level proxy semantics to obtain the query feature representation. Since the category label in the query set video is unknown, the embodiment of the present application performs action recognition on the query feature representation and the category prototype through the prototype matching module, so as to realize the matching of the second visual feature vector of the query set video with the category prototype of the support set video, so as to determine the action type of the second action video frame in the query set video, and realize the accurate recognition of the action type. The embodiment of the present application increases the diversity of semantic representation in terms of the richness and temporality of the semantic content, and gradually generates instance-level proxy semantic features with temporal dynamics through a multi-step pre-training knowledge guidance mechanism and a time-conditional back diffusion step, so as to realize the temporal semantic representation of the action video instance level, which can better capture the fine-grained semantic information implicit in the video, and jointly model this information with the visual features, thereby improving the overall generalization ability and robustness of the small sample action recognition system.

[0057] Reference Figure 1 , Figure 1 It is a specific method flow chart of a small sample action recognition method provided by an embodiment of the present application. The small sample action recognition method includes but is not limited to the following steps S101 to S105.

[0058] Step S101, obtaining a support set video, a query set video and a category label template, wherein the support set video includes multiple frames of first action video frames, the query set video includes multiple frames of second action video frames, and the category label template stores action category names corresponding to the first action video frames.

[0059] In step S101 of some embodiments, the instance-level semantic information is reflected in the differences in details such as the action subject, object and scene of each video. For example, for the same class name "climbing", different video descriptions: "a child climbing a mountain" and "an old man climbing stairs" are different. It is also reflected in the dynamic changes of the action at different moments. For example, a "high jump" action has various stages such as "run-up", "take-off", "ascent", "jumping down" and "landing". These fine-grained semantics are often unable to be conveyed through simple action class labels. In order to solve the above problems, the embodiment of the present application first obtains the support set video, the query set video and the category label template. The support set video includes multiple frames of the first action video frame, and the query set video includes multiple frames of the second action video frame, and obtains the category label template. The category label template stores the action category name corresponding to the first action video frame. For example, when the first action video frame is a person running, the category label template stores the action category name of running, which is convenient for the subsequent matching of the first action frame in the support set video and the category label template.

[0060] It can be understood that the category label template in the embodiment of the present application includes the action category name corresponding to the first action video frame, and each action category name is provided with its corresponding label, which can be words, sub-words or characters, etc. For example, [CLS] represents class, that is, a label, and also an action category name, and the embodiment of the present application does not make specific limitations.

[0061] Step S102, feature processing is performed on the support set video, the query set video and the category label template through a preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, wherein the multimodal training model includes an agent semantic diffusion module, a feature fusion module and a prototype matching module.

[0062] In step S102 of some embodiments, since some fine-grained semantics often cannot be conveyed through simple action class labels, the embodiments of the present application perform feature processing on the support set video, the query set video and the category label template through a preset multimodal training model to obtain the first visual feature vector corresponding to each first action video frame, the class-level semantic feature and the second visual feature vector corresponding to each second action video frame, thereby realizing the extraction of class-level semantic features, which facilitates the subsequent matching of video frames and action categories.

[0063] It can be understood that the multimodal training model in the embodiment of the present application will perform feature processing on each first action video frame in the support set video to obtain a first visual feature vector corresponding to each first action video frame. Similarly, the multimodal training model will also perform feature processing on each second action video frame in the query set video to obtain a second visual feature vector corresponding to each second action video frame.

[0064] Step S103, input the first visual feature vector, the class-level semantic features and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing, and output the first instance-level proxy semantics corresponding to the first visual feature vector and the second instance-level proxy semantics corresponding to the second visual feature vector.

[0065] In step S103 of some embodiments, the first visual feature vector, class-level semantic features, and second visual feature vector are input into a proxy semantic diffusion module for noise diffusion processing, and instance-level proxy semantic features with temporal dynamics are gradually generated through a multi-step pre-training knowledge guidance mechanism and a time-conditional back diffusion step. The first instance-level proxy semantics corresponding to the first visual feature vector and the second instance-level proxy semantics corresponding to the second visual feature vector are output, thereby realizing a temporal semantic representation of the action video instance level, which can better capture the fine-grained semantic information implicit in the video, and jointly model this information with the visual features, thereby improving the overall generalization ability and robustness of the small sample action recognition system.

[0066] It is worth noting that in the process of noise diffusion processing of the first visual feature vector, the class-level semantic features, and the second visual feature vector input into the proxy semantic diffusion module, the embodiment of the present application uses a multi-step pre-training knowledge guidance mechanism to mine the instance-level semantics inherent in the video, and gradually improves the quality of the semantics during the entire reverse diffusion process. The temporal conditional reverse diffusion step generates a video proxy semantics with temporal dynamics through a temporal conditional interpolator combined with reverse diffusion.

[0067] Step S104, feature fusion module is used to perform feature representation on the first instance-level proxy semantics, class-level semantic features and the first visual feature vector to construct a category prototype corresponding to the support set video, and feature fusion module is used to perform feature fusion on the second instance-level proxy semantics to obtain query feature representation.

[0068] In step S104 of some embodiments, the first instance-level proxy semantics, class-level semantic features and the first visual feature vector are represented by a feature fusion module, and frame-by-frame semantics and visual knowledge are fused to enhance the semantic expression capability in a single-frame video. At the same time, a temporal relationship between different frames is established. The second instance-level proxy semantics are fused by a feature fusion module to obtain a query feature representation, which facilitates subsequent matching of corresponding action labels for the query set video and improves matching accuracy.

[0069] Step S105 , performing action recognition on the query feature representation and the category prototype through a prototype matching module to determine the action type of the second action video frame in the query set video.

[0070] In step S105 of some embodiments, action recognition is performed on the query feature representation and the category prototype through the prototype matching module to achieve accurate recognition of the action category in the query set video. The entire process does not require a large amount of labeled data. By jointly modeling the support set video and the visual features, the overall generalization ability of the small sample action recognition system is improved.

[0071] Reference Figure 2 , Figure 2 It is a flowchart of a specific method for performing feature processing on support set videos, query set videos and category label templates through a preset multimodal training model provided by an embodiment of the present application. The method includes but is not limited to the following steps S201 to S202.

[0072] It should be noted that the multimodal training model includes a shared visual encoder and a text encoder.

[0073] Step S201, input the support set video into a preset multimodal training model, so that the shared visual encoder extracts features from the first action video frame in the support set video, outputs a first visual feature vector corresponding to each first action video frame, and enables the text encoder to extract features from the category label template and output class-level semantic features.

[0074] In step S201 of some embodiments, in the process of performing feature processing on the support set video, the query set video and the category label template through a preset multimodal training model, the embodiment of the present application first inputs the support set video into the preset multimodal training model so that the shared visual encoder extracts features from the first action video frame in the support set video. Specifically, the multimodal training model converts the first action video frame into an image and divides it into blocks of a fixed size. Then, each image block is mapped to a feature space of a fixed dimension through a trainable linear projection layer. In order to retain the spatial relationship of the image blocks, position codes are added to the feature vector of each image block. Then, these image block features with position codes are sent to the Transformer model. The Transformer model processes these features through a self-attention mechanism to capture the relationship between image blocks and the global structure of the image. Finally, the feature vector processed by the Transformer is extracted as the visual feature vector of the video frame, and the first visual feature vector corresponding to each first action video frame is output, so that the frame information of the video content can be captured, which is convenient for subsequent feature matching. At the same time, the text encoder extracts features from the category label template. Specifically, it converts the tag sequence in the category label template into an embedding vector to capture the semantic information of the vocabulary. The text encoder learns the internal structure and relationship of the text through the self-attention mechanism, and then takes out the output vector corresponding to the [CLS] tag. This vector is designed to represent the comprehensive features of the entire input sequence, that is, the class-level semantic features of the category label, thereby realizing the extraction of class-level semantic features, which is convenient for the subsequent matching of video frames and action categories.

[0075] It can be understood that the shared visual encoder in the embodiment of the present application adopts the pre-trained CLIP ViT-B / 16 model.

[0076] Step S202: input the query set video into a preset multimodal training model so that the shared visual encoder extracts features from the second action video frames of the query set video and outputs a second visual feature vector corresponding to each second action video frame.

[0077] In step S202 of some embodiments, the query set video is input into a preset multimodal training model so that the shared visual encoder extracts features from the second action video frame of the query set video. Specifically, the specific process of processing the second action video frame in the embodiment of the present application is consistent with the process of processing the first action video frame, which will not be repeated here. After the shared visual encoder extracts the features of the second action video frame, a second visual feature vector corresponding to each second action video frame is output to achieve the extraction of feature information in the query set video and improve the accuracy of action identification.

[0078] Reference Figure 3 , Figure 3 This is a flowchart of a specific method provided by an embodiment of the present application, in which a first visual feature vector, a class-level semantic feature, and a second visual feature vector are input into an agent semantic diffusion module for noise diffusion processing. The small sample action recognition method includes but is not limited to the following steps S301 to S303.

[0079] It should be noted that the semantic diffusion module includes a first semantic diffusion submodule and a second semantic diffusion submodule. The first semantic diffusion submodule is used to process class-level semantic features and a first visual feature vector, and the second semantic diffusion submodule is used to process a second visual feature vector.

[0080] Step S301 : gradually diffuse the second visual feature vector through a second semantic diffusion submodule to obtain second instance-level proxy semantics.

[0081] In step S301 of some embodiments, the second visual feature vector is gradually diffused through the second semantic diffusion sub-module, and the second instance-level proxy semantics is generated through a progressive K-step diffusion process, thereby capturing richer dynamic semantic information than the original category-level semantics, further improving the accuracy of action recognition.

[0082] Step S302: forward diffuse the class-level semantic features through the first semantic diffusion submodule to obtain noisy semantic features and embedded representations.

[0083] In step S302 of some embodiments, the class-level semantic features are forward diffused by the first semantic diffusion submodule. Specifically, the first semantic diffusion submodule adds noise to the class-level semantic features by a Markov chain that gradually adds noise. ) is converted into a series of latent representations with increasing noise levels , and each diffusion step corresponds to an embedding representation , which embeds The noisy semantic features and embedded representations are obtained by initializing with a multilayer perceptron (MLP).

[0084] In some embodiments, the noisy semantic feature is calculated by the following formula:

[0085] ;

[0086] in, It is k The noise factor of the step, It is a class-level semantic feature; N(0,1) is the noise that follows the standard normal distribution.

[0087] It is understandable that at each step In the above example, the category-level semantic features Different degrees of noise will be added to generate noisy features, that is, noisy semantic features , becomes one of the input conditions in the reverse denoising process, and increases monotonically with the steps, which means that the larger k is, The higher the degree of noise.

[0088] Step S303: input the first visual feature vector, the noisy semantic feature and the embedded representation into a first semantic diffusion submodule for reverse diffusion, and output a first instance-level proxy semantics.

[0089] In step S303 of some embodiments, the first visual feature vector, the noisy semantic features and the embedded representation are input into the first semantic diffusion submodule for reverse diffusion. Specifically, the first semantic diffusion submodule in the embodiment of the present application reversely diffuses the first visual feature vector, the noisy semantic features and the embedded representation through a denoising neural network, and gradually reversely predicts the semantic features of the previous step k-1 from the current diffusion step k, and iterates K steps in sequence, and finally generates a first instance-level proxy semantics rich in semantic diversity, which can extract rich semantic elements in the visual content, thereby improving the accuracy of action recognition. The embodiment of the present application uses a multi-step pre-training knowledge guidance mechanism to mine the instance-level semantics inherent in the video, and gradually improves the quality of the semantics in the entire reverse diffusion process, so as to facilitate the subsequent recognition of actions in the query set video.

[0090] In some embodiments, the first instance-level proxy semantics is calculated by the following formula:

[0091] ;

[0092] in, It is the kth step Intermediate instance-level proxy semantics of frames, v t It is the first video in the support set. The first visual feature vector of the frame, σ k is the noise standard deviation of the k-th step of the back diffusion process, Z N(0,I) is random noise, is a temporal interpolator, which is used to capture the temporal changes of the first instance-level proxy semantics.

[0093] It is worth noting that the back diffusion of the embodiment of the present application gradually generates the proxy semantic features of each frame through denoising and introduces a time interpolator , adding timing information, and by combining the conditional input of the time dimension, the generated agent semantic features can capture the temporal dynamic changes of the action, thus solving the limitations of static semantic expression in the current method. , so that the generation process of agent semantics can capture dynamic features that change over time, thereby generating a series of semantic representations that evolve over time, enhancing the ability of agent semantics in capturing dynamic changes in actions and improving the accuracy of capturing dynamic actions.

[0094] In some embodiments, the time interpolator performs a K-step reverse recursive loop, and the output finally generated by the first semantic diffusion submodule is a proxy semantic feature s= ,in, Indicates the number of video frames, each Indicates time The ultimate goal of the first semantic diffusion submodule is to generate corresponding dynamic semantics for each local moment of the video, thereby surpassing the traditional static category-level semantics in the temporal dimension and providing richer semantic expressions for action recognition.

[0095] Reference Figure 4 , Figure 4 This is a flowchart of a specific method for performing feature representation of first instance-level proxy semantics, class-level semantic features, and a first visual feature vector through a feature fusion module provided in an embodiment of the present application. The method includes but is not limited to the following steps S401 to S403.

[0096] It should be noted that the feature fusion module includes an intra-frame fusion layer and an inter-frame fusion layer.

[0097] Step S401 : input the first instance-level proxy semantics, the class-level semantic features and the first visual feature vector into an intra-frame fusion layer for visual semantic feature fusion, and output frame information.

[0098] In step S401 of some embodiments, the first instance-level proxy semantics, class-level semantic features, and the first visual feature vector are input into the intra-frame fusion layer for visual semantic feature fusion. The intra-frame fusion layer fuses the frame-by-frame semantics with the visual knowledge and outputs the frame information. The intra-frame fusion layer enhances the semantic expression capability in a single-frame video by combining visual and semantic features.

[0099] Step S402: input the frame information into the inter-frame fusion layer, so that the inter-frame fusion layer establishes a temporal relationship between different frames according to the frame information and outputs a feature representation.

[0100] In step S402 of some embodiments, the frame information is input into the inter-frame fusion layer, so that the inter-frame fusion layer establishes the temporal relationship between different frames according to the frame information, generates the final feature representation by combining the multi-frame information, and outputs the feature representation. By fusing the multi-frame information, the dynamic changes and motion information in the video can be captured, thereby enhancing the model's understanding of the video content, and through the inter-frame fusion layer, the model can perform temporal reasoning and understand the changing relationship of objects in the time domain.

[0101] Step S403, performing an average operation on the feature representations of the same category in the support set video to construct a category prototype corresponding to the support set video.

[0102] In step S403 of some embodiments, a mean operation is performed on the feature representations of the same category in the support set video to construct a category prototype corresponding to the support set video, wherein the category prototype can represent a typical feature vector or feature set of a category, thereby facilitating the subsequent alignment of features of different modalities in the multimodal space.

[0103] Reference Figure 5 , Figure 5 It is a flowchart of a specific method for performing feature fusion on second instance-level proxy semantics through a feature fusion module provided in an embodiment of the present application. The method includes but is not limited to the following steps S501 to S502.

[0104] Step S501 , fusing the second instance-level proxy semantics frame by frame through an intra-frame fusion layer to obtain a fusion result.

[0105] Step S502: establishing a temporal relationship between different frames in the fusion result through an inter-frame fusion layer to obtain a query feature representation.

[0106] In some embodiments, in step S501 to step S502, in the process of feature fusion of the second instance-level proxy semantics through the feature fusion module, the second instance-level proxy semantics are first fused frame by frame through the intra-frame fusion layer to obtain a fusion result. The frame-by-frame fusion can capture the dynamic changes and motion information in the video, thereby enhancing the model's understanding of the video content and reducing the recognition errors caused by the noise or incompleteness of a single frame. Then, the temporal relationship between different frames in the fusion result is established through the inter-frame fusion layer to reduce the recognition errors caused by the noise or incompleteness of a single frame, understand the change relationship of the object in the time domain, and obtain the query feature representation, so as to facilitate the subsequent matching of the corresponding action labels for the query set video and improve the matching accuracy.

[0107] Reference Figure 6 , Figure 6This is a flowchart of a specific method for performing action recognition on query feature representation and category prototypes through a prototype matching module provided in an embodiment of the present application. The method includes but is not limited to the following steps S601 to S602.

[0108] Step S601: input the query feature representation and the category prototype into a prototype matching module for similarity matching to obtain a matching result.

[0109] Step S602: performing action recognition on the second action video frame in the query set video according to the matching result to determine the action type of the second action video frame in the query set video.

[0110] In steps S601 to S602 of some embodiments, in the process of performing action recognition on the query feature representation and the category prototype through the prototype matching module, since the category label in the query set is unknown, the input of the intra-frame fusion layer does not contain semantic features. In order to improve the matching accuracy of the video frame and the semantic feature, the embodiment of the present application inputs the query feature representation and the category prototype into the prototype matching module for similarity matching. Specifically, the similarity between the query feature representation and each category prototype is calculated. The similarity between the query feature representation and each category prototype can be calculated by methods such as Euclidean distance and cosine similarity, and then the similarity is converted into a probability distribution through a softmax function to represent the probability that the query sample belongs to each category. Finally, the category with the maximum probability is selected as the predicted category of the query sample according to the probability distribution to obtain a matching result. Thereafter, action recognition is performed on the second action video frame in the query set video according to the matching result, and the action type of the second action video frame in the query set video is determined, so as to achieve accurate recognition of the action category in the query set video. The entire process does not require a large amount of labeled data. By jointly modeling the support set video and the visual features, the overall generalization ability of the small sample action recognition system is improved.

[0111] Reference Figure 7 , Figure 7 This is a specific method flow chart of a small sample action recognition method provided by another embodiment of the present application, and the method includes but is not limited to the following steps S701.

[0112] It should be noted that step S701 occurs after the first visual feature vector, class-level semantic features and the second visual feature vector are input into the proxy semantic diffusion module for noise diffusion processing, and the first instance-level proxy semantics corresponding to the first visual feature vector and the second instance-level proxy semantics corresponding to the second visual feature vector are output.

[0113] Step S701, training the proxy semantic diffusion module through a preset supervision loss function.

[0114] In step S701 of some embodiments, in order to ensure the semantic correlation between the generated proxy semantic features and the visual features, the embodiments of the present application train the proxy semantic diffusion module through a preset supervised loss function. Specifically, in the back-diffusion process, each step uses CLIP (Contrastive Language–Image Pre-training) cosine similarity to optimize the correlation between the instance-level proxy semantics and the visual feature vector, thereby achieving continuous improvement in the quality of the proxy semantics.

[0115] Specifically, the supervision loss function in the embodiment of the present application is expressed as follows:

[0116] ;

[0117] Among them, τ is the temperature coefficient defined in CLIP, which is used to adjust the sensitivity of the model to the similarity between samples; represents cosine similarity; i is a positive sample, B is the batch size; Represents instance-level proxy semantics, Represents the visual feature vector.

[0118] It is worth noting that CLIP guidance is performed at each step of the denoising process, not just the last step, ensuring that the proxy semantic generation process is always consistent with the visual features and avoiding directional bias in long chain generation.

[0119] In order to more clearly illustrate the above-mentioned small sample action recognition method, a specific example is given below for illustration.

[0120] Example 1:

[0121] The limitation of existing methods in feature modeling is that they rely too much on visual modalities or category-level semantics and fail to fully utilize the rich semantic information at the instance level, which limits the performance of the model in complex action recognition tasks. In small-sample action recognition tasks, relying solely on the semantic information of class names is often insufficient. How to capture and model these instance-level semantic details and temporal information has become a key challenge to improve model performance.

[0122] The embodiment of the present application proposes a small sample action recognition method based on progressive learning of instance-level proxy semantics (ProLIPS). Without introducing external semantics, the method gradually learns instance-level temporal semantic information from the video data itself through a multi-step pre-training knowledge guidance mechanism and a time-conditional back diffusion step, thereby expanding and enriching the original video label semantics. Compared with the static semantics of the prior art that relies on unified class name labels, its innovation lies in the realization of temporal semantic representation of action video instance level, and the joint modeling of these temporal semantic representations with visual features, thereby improving the overall generalization ability and robustness of the small sample action recognition system.

[0123] Reference Figure 8 , Figure 8 This is a schematic diagram of the model architecture of a multimodal training model provided in an example of the present application.

[0124] In some embodiments, the multimodal training model mainly includes a visual encoding and class-level semantic feature extraction module, a proxy semantic diffusion (PSD) module, a feature fusion module, and a prototype matching module. Among them, the core component is the proxy semantic diffusion module, which is used to generate instance-level proxy semantic features with temporal dynamics that correspond to video instances one by one.

[0125] Specifically, PSD is learned using a multi-step pre-trained knowledge guidance mechanism (using pre-trained knowledge from the CLIP model through multi-step guidance) and a temporal conditional back-diffusion step. The multi-step pre-trained knowledge guidance mechanism is used to mine the instance-level semantics inherent in the video and gradually improve the quality of the semantics throughout the back-diffusion process. The temporal conditional back-diffusion step generates video proxy semantics with temporal dynamics through a temporal conditional interpolator combined with back-diffusion.

[0126] The following will introduce the functions and specific workflow of each component step by step:

[0127] In some embodiments, the visual encoding and class-level semantic feature extraction module is used to extract the first visual feature vector and class-level semantic features of the support set video, and is also used to extract the second visual feature vector of the query set video. In each training and testing episode, there are two input branches, which input the support set video and the query set video respectively. First, the video data of the support set video and the query set video are processed by a shared visual encoder (using the pre-trained CLIP ViT-B / 16 model) to generate a first visual feature vector corresponding to the support set video and a second visual feature vector corresponding to the query set video, which are expressed as ,in is the batch size, is the sampling frame number, is the feature dimension.

[0128] In the support set video branch, additional class-level semantic features are extracted Class-level semantic features are obtained through the CLIP text encoder Processing category label templates (such as "a photo of [CLS]") to generate ([CLS] represents class, i.e., label, and also the name of the action category). In order to maintain the transferable knowledge of the pre-trained model, the present embodiment of the application fixes the text encoder Only the visual encoder is updated part of the layer.

[0129] In some embodiments, the core of ProLIPS is the Proxy Semantic Diffusion (PSD) module, which receives instance-level video features. (first visual feature vector and second visual feature vector) and category-level semantic features, and generates instance-level proxy semantics through a progressive K-step diffusion process. The embodiment of the present application proposes a novel progressive instance-level proxy semantics learning framework (ProLIPS) to solve the problem of insufficient semantic diversity in small sample action recognition tasks. By generating instance-level proxy semantics, the framework can capture richer dynamic semantic information than the original category-level semantics, thereby improving the accuracy of action recognition.

[0130] Specifically, the proxy semantic diffusion module is used to perform noise diffusion processing on the first visual feature vector, the class-level semantic feature, and the second visual feature vector, which mainly includes the following steps:

[0131] Forward diffusion is a Markov chain that gradually adds noise, the goal is to transform the class-level semantic features (represented as ) is converted into a series of latent representations with increasing noise levels , and each diffusion step corresponds to an embedding representation, which is initialized by a multilayer perceptron (MLP) to obtain noisy semantic features and embedded representations.

[0132] In some embodiments, the noisy semantic feature is calculated by the following formula:

[0133] ;

[0134] in, It is k The noise factor of the step, It is a class-level semantic feature; N(0,1) is the noise that follows the standard normal distribution.

[0135] At every step In the above example, the category-level semantic features Different degrees of noise will be added to generate noisy features, that is, noisy semantic features , becomes one of the input conditions in the reverse denoising process, and increases monotonically with the steps, which means that the larger k is, In addition, each diffusion step corresponds to an embedding representation , which embeds Initialized by a multi-layer perceptron.

[0136] In the back-diffusion stage, the PSD module uses instance-level video features , category-level noisy semantic features and k Step Embedding As input conditions, generate agent semantic features This process is iterated in reverse K step, and finally generate instance-level proxy semantic features rich in semantic diversity.

[0137] Reference Fig. 9 , Fig. 9 It is a schematic diagram of the reverse diffusion process provided in an example of the present application.

[0138] like Fig. 9 As shown, the back diffusion process is implemented through a parameterized denoising neural network , gradually reversely predict the semantic features of the previous step k-1 from the current diffusion step k, iterate K steps in sequence, and finally generate the first instance-level proxy semantics rich in semantic diversity. The reverse diffusion process generates the intermediate proxy semantics {s k ,s k−1 ,…,s 0}.

[0139] The instance-level proxy semantics is calculated using the following formula:

[0140] ;

[0141] in, It is the kth step Intermediate instance-level proxy semantics of frames, v t It is the first video in the support set. The first visual feature vector of the frame, σ k is the noise standard deviation of the k-th step of the back diffusion process, Z N(0,I) is random noise, is a temporal interpolator, which is used to capture the temporal changes of the first instance-level proxy semantics.

[0142] Back diffusion gradually generates proxy semantic features for each frame through denoising and introduces a temporal interpolator ,By combining the conditional input of the time dimension, the generated agent semantic features can capture the temporal dynamic changes of the action.

[0143] In order to solve the limitation of static semantic expression in existing methods, the present invention further introduces a random time interpolator , by combining the conditional input of the time dimension , so that the generation process of agent semantics can capture dynamic features that change over time, thereby generating a series of semantic representations that evolve over time. This improvement significantly enhances the ability of agent semantics to capture dynamic changes in actions.

[0144] The time interpolator goes through a K-step reverse recursive loop, and the output generated by the first semantic diffusion submodule is a proxy semantic feature s= ,in, Indicates the number of video frames, each Indicates time The ultimate goal of the first semantic diffusion submodule is to generate corresponding dynamic semantics for each local moment of the video, thereby surpassing the traditional static category-level semantics in the temporal dimension and providing richer semantic expressions for action recognition.

[0145] In order to ensure the semantic relevance of the generated proxy semantic features and visual features, the framework uses a pre-trained knowledge guidance mechanism for supervision. In the back-diffusion process, each step optimizes the proxy semantics through CLIP cosine similarity. With instance-level video features This mechanism can continuously improve the quality of agent semantics.

[0146] Specifically, the supervised loss function Lg of CLIP is defined as:

[0147] ;

[0148] Among them, τ is the temperature coefficient defined in CLIP, which is used to adjust the sensitivity of the model to the similarity between samples; represents cosine similarity; i is a positive sample, B is the batch size; Represents instance-level proxy semantics, Represents the visual feature vector.

[0149] It is worth noting that CLIP guidance is performed at each step of the denoising process, not just the last step, ensuring that the proxy semantic generation process is always consistent with the visual features and avoiding directional bias in long chain generation.

[0150] In some embodiments, the feature fusion module is used to fuse instance-level proxy semantics, class-level semantic features, and visual feature vectors. Specifically, the generated instance-level proxy semantics Class-level semantic features , visual feature vector They are then input into the intra-frame fusion layer to fuse frame-by-frame semantic and visual knowledge.

[0151] The intra-frame fusion layer enhances the semantic expression capability in a single frame of video by combining visual and semantic features. After that, the inter-frame fusion layer is responsible for establishing the temporal relationship between different frames and combining multi-frame information to generate the final feature representation. The final features of the same category are averaged to construct the category prototype of the support set.

[0152] In some embodiments, the prototype matching module is used to match the category prototype of the support set video with the second video action frame in the query set video. In the query set video branch, since the category label in the query set video is unknown, the input of the intra-frame fusion layer does not contain class-level semantic features. Finally, the feature representation of the query video is matched with the support set prototypes in the prototype space to complete the action classification task.

[0153] It should be noted that the limitation of existing methods in feature modeling is the excessive reliance on visual modality or category-level semantics, and the failure to make full use of the rich semantic information at the instance level, which limits the performance of the model in the face of complex action recognition tasks. In small sample action recognition tasks, the semantic information that relies solely on the class name is often insufficient. How to capture and model these instance-level semantic details and timing information has become a key challenge to improve model performance. In response to the above problems, the embodiments of the present application aim to increase the diversity of semantic representation from the richness and timing of semantic content. Its core component PSD, i.e., Proxy Semantic Diffusion, gradually generates instance-level proxy semantic features with temporal dynamics through a multi-step pre-training knowledge guidance mechanism and a time-conditional reverse diffusion step. The progressive learning of instance-level proxy semantics helps to achieve the temporal semantic representation of the action video instance level, which can better capture the fine-grained semantic information implicit in the video, and jointly model this information with visual features, thereby improving the overall generalization ability and robustness of the small sample action recognition system.

[0154] Reference Fig.10 , Fig.10This is a performance comparison diagram of the ProLIPS framework provided in an example of this application on the SSv2-Full, Kinetics, and UCF101 datasets. Fig.11 , Fig.11 This is a performance comparison diagram of the ProLIPS framework provided in an example of this application on the SSv2-Small and HMDB51 datasets.

[0155] Depend on Fig.10 and Fig.11 It can be seen that the ProLIPS framework has been tested on multiple commonly used few-shot action recognition datasets, such as SSv2-Full, SSv2-Small, Kinetics, UCF101, and HMDB51, and compared with current competitive methods. The results show that ProLIPS has a significant performance improvement over existing baseline methods. For example, in the 1-shot and 5-shot settings of the SSv2-Full dataset, ProLIPS (using the CLIP-ViT-B / 16 backbone network) is 6.0% and 3.2% higher than the HyRSM baseline, respectively. This shows that by modeling instance-level semantic information, ProLIPS can significantly improve the model's action recognition accuracy.

[0156] In addition, ProLIPS is particularly advantageous in low-sample scenarios, especially in 1-shot tasks. For example, in Kinetics' 1-shot tasks, ProLIPS is 4.4% higher than the HyRSM baseline. This shows that semantic information is particularly critical when visual data is scarce. ProLIPS exhibits strong adaptability in low-sample tasks by progressively generating instance-level semantic information. Compared with the most advanced methods such as CLIP-FSAR and CLIP-CPM2C, ProLIPS exceeds them by 2.3% and 3.1% in 1-shot tasks on the SSv2-Full dataset, respectively. Compared with other datasets, the method of the embodiment of the present application has produced greater improvements on the SSv2 dataset. This can be attributed to the incomplete semantic information of text labels in the SSv2 dataset, which usually lacks subject entities. Therefore, ProLIPS of the embodiment of the present application can combine time to dynamically mine rich instance-level semantic information, especially in scenarios where semantic label information is incomplete, showing strong robustness and adaptability. This is where the CLIP-FSAR and CLIP-CPM2C methods that rely only on data label semantics are insufficient.

[0157] Reference Fig.12, taking the control processor 1001 and memory 1002 in the electronic device 1000 as an example that can be connected via a bus. The memory 1002, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory 1002 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk memory, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1002 may optionally include a memory remotely arranged relative to the control processor 1001, and these remote memories may be connected to the electronic device 1000 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] Those skilled in the art will understand that Fig.12 The device structure shown in the figure does not constitute a limitation on the electronic device 1000, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0159] The embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by one or more control processors, for example, Fig.12 Execution by one of the control processors 1001 may enable the one or more control processors to execute the small sample action recognition method in the above method embodiment.

[0160] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

Claims

1. A small sample action recognition method, characterized in that: include: Obtain a support set video, a query set video, and a category label template, wherein the support set video includes multiple frames of first action video frames, the query set video includes multiple frames of second action video frames, and the category label template stores an action category name corresponding to the first action video frame; The support set video, the query set video and the category label template are feature processed by a preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, wherein the multimodal training model includes a proxy semantic diffusion module, a feature fusion module and a prototype matching module; Inputting the first visual feature vector, the class-level semantic feature, and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing, and outputting a first instance-level proxy semantic corresponding to the first visual feature vector and a second instance-level proxy semantic corresponding to the second visual feature vector; Performing feature representation on the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector through the feature fusion module to construct a category prototype corresponding to the support set video, and performing feature fusion on the second instance-level proxy semantics through the feature fusion module to obtain a query feature representation; Performing action recognition on the query feature representation and the category prototype by the prototype matching module to determine the action type of the second action video frame in the query set video; The semantic diffusion module includes a first semantic diffusion submodule and a second semantic diffusion submodule; the first visual feature vector, the class-level semantic feature and the second visual feature vector are input into the proxy semantic diffusion module for noise diffusion processing, and the first instance-level proxy semantics corresponding to the first visual feature vector and the second instance-level proxy semantics corresponding to the second visual feature vector are output, including: The second visual feature vector is gradually diffused by the second semantic diffusion submodule to obtain second instance-level proxy semantics; Forward diffusion of the class-level semantic features is performed through the first semantic diffusion submodule to obtain noisy semantic features and embedded representations; Inputting the first visual feature vector, the noisy semantic feature and the embedded representation into the first semantic diffusion submodule for reverse diffusion, and outputting first instance-level proxy semantics; The first instance-level proxy semantics is calculated by the following formula: ； in, is the intermediate instance-level proxy semantics of the k-th step and the t-th frame, is the first visual feature vector of the tth frame in the support set video, σ k is the noise standard deviation of the k-th step of the back diffusion process, Z N(0,I) is random noise, is a denoising neural network, It is k The noise factor of the step, is a time interpolator used to capture the temporal changes of the first instance-level proxy semantics.

2. The small sample action recognition method according to claim 1, characterized in that: The multimodal training model includes a shared visual encoder and a text encoder; the support set video, the query set video and the category label template are feature processed by the preset multimodal training model to obtain a first visual feature vector corresponding to each first action video frame, a class-level semantic feature and a second visual feature vector corresponding to each second action video frame, including: Input the support set video into a preset multimodal training model, so that the shared visual encoder performs feature extraction on the first action video frame in the support set video, outputs a first visual feature vector corresponding to each first action video frame, and the text encoder performs feature extraction on the category label template, and outputs a class-level semantic feature; The query set video is input into a preset multimodal training model so that the shared visual encoder extracts features from the second action video frames of the query set video and outputs a second visual feature vector corresponding to each second action video frame.

3. The small sample action recognition method according to claim 1, characterized in that: The feature fusion module includes an intra-frame fusion layer and an inter-frame fusion layer; the feature fusion module performs feature representation on the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector to construct a category prototype corresponding to the support set video, including: Inputting the first instance-level proxy semantics, the class-level semantic features, and the first visual feature vector into the intra-frame fusion layer for visual semantic feature fusion, and outputting frame information; Inputting the frame information into the inter-frame fusion layer, so that the inter-frame fusion layer establishes a temporal relationship between different frames according to the frame information and outputs a feature representation; A mean operation is performed on feature representations of the same category in the support set video to construct a category prototype corresponding to the support set video.

4. The small sample action recognition method according to claim 3, characterized in that: The step of performing feature fusion on the second instance-level proxy semantics by the feature fusion module to obtain a query feature representation includes: fusing the second instance-level proxy semantics frame by frame through the intra-frame fusion layer to obtain a fusion result; The temporal relationship between different frames in the fusion result is established through the inter-frame fusion layer to obtain the query feature representation.

5. The small sample action recognition method according to claim 1, characterized in that: The performing action recognition on the query feature representation and the category prototype by the prototype matching module to determine the action type of the second action video frame in the query set video includes: Inputting the query feature representation and the category prototype into the prototype matching module for similarity matching to obtain a matching result; Perform action recognition on the second action video frame in the query set video according to the matching result to determine the action type of the second action video frame in the query set video.

6. The small sample action recognition method according to claim 1, characterized in that: After inputting the first visual feature vector, the class-level semantic feature, and the second visual feature vector into the proxy semantic diffusion module for noise diffusion processing, and outputting first instance-level proxy semantics corresponding to the first visual feature vector and second instance-level proxy semantics corresponding to the second visual feature vector, the method further includes: The agent semantic diffusion module is trained by a preset supervised loss function.

7. The small sample action recognition method according to claim 1, characterized in that: The noisy semantic feature is calculated by the following formula: ； in, It is k The noise factor of the step, It is a class-level semantic feature; N(0,1) is the noise that follows the standard normal distribution.

8. An electronic device, characterized in that: It includes at least one processor and a memory for communicating with the at least one processor; the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the small sample action recognition method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the small sample action recognition method as described in any one of claims 1 to 7.

Images