A few-shot action recognition method based on prototype enhancement and query alignment

Abstract

The application discloses a few-sample action recognition method based on prototype enhancement and query alignment, and belongs to the technical field of computer vision, comprising the following steps: constructing a support set, a query set and a text description of a corresponding action category according to an action video dataset; constructing an action recognition network model and training the model, wherein the model performs the following operations: inputting the query set and the support set into an image encoder to extract query clustering features and support clustering features, and inputting the text description into a text encoder to extract text features; obtaining a plurality of semantic enhancement prototypes and aligned query features according to the support clustering features, the query clustering features and all the text features; obtaining the final action recognition result of the corresponding video in the query set by using a classifier; and inputting the support set, the query set to be recognized and the text description of the corresponding action category into the trained action recognition network model to obtain the final action recognition result of the query set to be recognized. The application can improve the accuracy of small-sample action recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a few-shot action recognition method based on prototype enhancement and query alignment. Background Technology

[0002] Few-Shot Action Recognition (FSAR) aims to classify human actions in videos using only a limited number of labeled samples. Because collecting and labeling fine-grained video data is both time-consuming and expensive, the scarcity of such high-quality data makes large-scale, fully supervised training impractical in many real-world scenarios. Currently, using large-scale pre-trained models for few-shot action recognition has become the mainstream solution, but in situations of extreme data scarcity, they often struggle to accurately capture key video-specific information, such as complex temporal dynamics, exhibiting the following limitations:

[0003] First, the inherent temporal redundancy in videos often weakens key motion cues. Existing standard methods, such as global average pooling or heavy Transformer modeling, either lose fine-grained temporal structure or are prone to overfitting under small sample conditions, failing to effectively filter out irrelevant background frames.

[0004] Second, visual ambiguity arises from the inherent semantic gap between limited visual samples and abstract category labels. Most existing methods rely on static feature extraction followed by shallow similarity matching. This failure to utilize textual semantics to correct visual features during the encoding stage leads to misclassification when different actions share similar visual backgrounds. Summary of the Invention

[0005] The purpose of this invention is to address the above-mentioned problems by proposing a few-sample action recognition method based on prototype enhancement and query alignment, which overcomes the difficulties in obtaining batch samples and the shortcomings in handling temporal redundancy and visual-semantic misalignment in the prior art, thereby improving the accuracy of few-sample action recognition.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention proposes a few-shot action recognition method based on prototype enhancement and query alignment, comprising the following steps:

[0008] S1. Construct a support set, a query set, and text descriptions of corresponding action categories based on the motion video dataset;

[0009] S2. Construct an action recognition network model and train it by inputting a support set, a query set, and text descriptions of the corresponding action categories. The action recognition network model includes an image encoder, a text encoder, a cross-modal prototype enhancement module, a text adaptive query alignment module, and a classifier, and is used to perform the following operations:

[0010] S21. Input the query set and support set into the image encoder to extract the query clustering features and support clustering features respectively. Input the text description of each action category into the text encoder to extract the corresponding text features.

[0011] S22. Input clustering features and all text features into the cross-modal prototype enhancement module to obtain... A semantic enhancement prototype is generated by inputting query clustering features and all text features into a text adaptive query alignment module. Alignment query features, The number of action categories;

[0012] S23. Use a classifier to calculate the similarity between all aligned query features and semantically enhanced prototypes of each video in the query set, and select the action category corresponding to the semantically enhanced prototype with the highest similarity under the same video as the final action recognition result of the corresponding video in the query set.

[0013] S3. Input the support set, the query set to be identified, and the text description of the corresponding action category into the trained action recognition network model to obtain the final action recognition result of the query set to be identified.

[0014] Preferably, a support set, a query set, and text descriptions of corresponding action categories are constructed based on the action video dataset, as follows:

[0015] S11. Select from the motion video dataset There are 10 action categories, and the corresponding video for each action category is extracted. A video, forming a collection Support set of videos and corresponding action category tags;

[0016] S12, from Extract at least one video from the videos corresponding to each action category to form a query set;

[0017] S13. Utilizing large language models to... Each action category is described in text, including the start phase, the process phase, and the end phase.

[0018] Preferably, the image encoder performs the following operations on each video in the query set or support set:

[0019] S211. Encode the video to obtain the corresponding frame-level feature sequence, which contains the visual embedding sequence of several video frames.

[0020] S212. Calculate the similarity between the visual embedding features of any two adjacent video frames in the frame-level feature sequence;

[0021] S213. Merge the visual embedding features of the two adjacent video frames with the highest similarity into a new visual embedding feature. The merging method is to calculate the mean or to use a weighted average.

[0022] S214. Replace the visual embedding features of the two adjacent video frames with the highest similarity with new visual embedding features to update the frame-level feature sequence, and record the length of the updated frame-level feature sequence.

[0023] S215. Return to step S212 until the length of the frame-level feature sequence reaches a preset number. The final frame-level feature sequence is obtained as the clustering feature of the corresponding video. The clustering feature contains If a time-series segment feature is defined, then the clustering features of all videos in the query set are used to form the query clustering features, and the clustering features of all videos in the support set are used to form the support clustering features.

[0024] Preferably, the similarity is calculated using cosine similarity.

[0025] Preferably, clustering features and all text features are input into the cross-modal prototype enhancement module to obtain... There are several semantic enhancement prototypes, each obtained as follows:

[0026] The features of each temporal segment of the current video in the clustering features are concatenated with the corresponding text features in the channel dimension to form the first mixed feature of the corresponding temporal segment features;

[0027] Map the first mixed feature of each time segment feature to the first query vector, and map each time segment feature of the current video to the first key vector and the first value vector respectively;

[0028] Calculate the first attention weight between the first query vector and the first key vector of the corresponding time segment feature. Multiply the first attention weight with the first value vector and then perform weighted aggregation with the first query vector of the corresponding time segment feature to form the first weighted feature.

[0029] The first transformed feature is obtained by using a feedforward neural network to extract features from the first weighted feature. The first transformed feature of the corresponding time segment feature is then residually connected with the first weighted feature and then normalized by layer to obtain the first aggregated feature of the corresponding time segment feature.

[0030] The first aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the semantically enhanced prototype of the corresponding video.

[0031] Preferably, the query clustering features and all text features are input into the text adaptive query alignment module to generate... The alignment query features are obtained as follows:

[0032] Build for the action category of the current video The semantic hypothesis, of which the first... The semantic assumption is that the current video belongs to the [number]th [semantic hypothesis]. Each action category ;

[0033] Under each semantic assumption, a cross-modal attention mechanism is used to obtain the corresponding aligned query features, thereby generating... Alignment query features based on semantic hypotheses are used as the corresponding video. Alignment query features.

[0034] Preferably, under each semantic hypothesis, the corresponding aligned query features are obtained using a cross-modal attention mechanism. The aligned query features for the k-th semantic hypothesis are then obtained as follows:

[0035] The features of each temporal segment of the current video in the query cluster features are concatenated with the text features of the kth action category in the channel dimension to form the second hybrid feature of the corresponding temporal segment features;

[0036] The second mixed feature of each time segment feature is mapped to the second query vector, and the features of each time segment of the current video are mapped to the second key vector and the second value vector respectively.

[0037] Calculate the second attention weight between the second query vector and the second key vector corresponding to the time segment feature. Multiply the second attention weight with the second value vector and then perform weighted aggregation with the second query vector corresponding to the time segment feature to form the second weighted feature.

[0038] The second transformed feature is obtained by using a feedforward neural network to extract features from the second weighted feature. The second transformed feature of the corresponding time segment feature is then residually connected with the second weighted feature and normalized by layer to obtain the second aggregated feature of the corresponding time segment feature.

[0039] The second aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the corresponding video. Alignment query features.

[0040] Preferably, a dual maximization strategy is adopted to select the action category corresponding to the semantically enhanced prototype with the highest similarity under the same video as the final action recognition result of the corresponding video in the query set.

[0041] Preferably, the dual maximization strategy satisfies the following formula:

[0042]

[0043]

[0044] In the formula, This represents the final action recognition result for the corresponding video. Indicates in Find the expression within the parentheses that maximizes its value. , Indicates in Find below The maximum value, Indicates the current video's... The alignment query feature and the first Similarity of semantically enhanced prototypes Indicates the current video's... Alignment query features, Indicates the first A semantically enhanced prototype This indicates the calculation of cosine similarity. This indicates a pooling operation. , .

[0045] Preferably, the text encoder is a CLIP model text encoder.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] This invention performs action recognition by constructing and training an action recognition network model. The action recognition network model includes an image encoder, a text encoder, a cross-modal prototype enhancement module, a text adaptive query alignment module, and a classifier. Specifically, the image encoder and text encoder extract video and text features and perform temporal clustering. For the support set, the cross-modal prototype enhancement module uses text features to calibrate visual features, effectively filtering out background noise and temporal redundancy from the videos in the support set. Denoising and focusing help ensure that the enhanced prototype captures specific motion patterns of robust action categories. For the query set, the text adaptive query alignment module employs a hypothesis-driven strategy, interacting the videos in the query set with text features from all action categories to generate multiple dynamic calibration features (alignment query features). Finally, the final action recognition result for the query set is determined by calculating similarity and using a dual maximization strategy. This solves the alignment difficulty caused by unlabeled videos in the query set, dynamically adjusting the alignment query features using the full action category text space, significantly improving intra-class compactness and inter-class separability, and achieving dynamic calibration. This invention can effectively solve the problems of temporal redundancy and visual ambiguity, and significantly improve the accuracy of action recognition in small samples. It outperforms existing methods in both 1-shot and 5-shot settings, and has high recognition accuracy, especially in action recognition tasks with complex temporal sequences. Attached Figure Description

[0048] Figure 1 This is a flowchart of the few-shot action recognition method based on prototype enhancement and query alignment of the present invention;

[0049] Figure 2 This is a schematic diagram of the action recognition network model of the present invention;

[0050] Figure 3 A visualization comparison of the t-SNE of the existing TRX model and the method of this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] It should be noted that when a component is referred to as being "connected" to another component, it can be directly connected to the other component or there may be an intervening component. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application.

[0053] This application proposes a few-shot action recognition method based on prototype enhancement and query alignment. The few-shot action recognition task is modeled as an N-way K-shot meta-learning problem. Here, N-way indicates that in each action recognition process, a set of K-shot samples needs to be generated from... Classification is performed among candidate action categories; K-shot means that for each candidate action category, only one action is provided. Annotated video samples are used as support samples for model adaptive learning. This method addresses the temporal redundancy and visual ambiguity problems existing in existing pre-trained vision-language models for few-shot action recognition tasks by designing a cross-modal prototype enhancement module (CMPR) and a text adaptive query alignment module (TAQA), which realizes deep interaction and dynamic calibration between visual features and text semantics.

[0054] like Figures 1-3 As shown, a few-shot action recognition method based on prototype enhancement and query alignment includes the following steps:

[0055] S1. Construct a support set, a query set, and text descriptions of the corresponding action categories based on the action video dataset.

[0056] In one embodiment, a support set, a query set, and text descriptions of corresponding action categories are constructed based on the action video dataset, as follows:

[0057] S11. Select from the motion video dataset There are 10 action categories, and the corresponding video for each action category is extracted. A video, forming a collection Support set of videos and corresponding action category tags;

[0058] S12, from Extract at least one video from the videos corresponding to each action category to form a query set;

[0059] S13. Utilizing large language models to... Each action category is described in text, including the start phase, the process phase, and the end phase.

[0060] The action video dataset contains videos from various application scenarios, including human daily behavior recognition, intelligent monitoring, sports analysis, human-computer interaction gesture recognition, video content retrieval, and industrial operation process monitoring. The corresponding action categories include walking, running, jumping, waving, bending over to pick up an object, falling, throwing, weightlifting, gesture operations, training actions, and assembly process videos. Based on the action video dataset, a support set, a query set, and corresponding text descriptions of the action categories are constructed, as follows:

[0061] Obtain the support set: Select from the motion video dataset There are 10 action categories, and the corresponding video for each action category is extracted. Using several videos as reference samples, a system is formed containing... Support sets for each video and its corresponding category tag , It is a positive integer.

[0062] Retrieving the Query Set: From At least one video is extracted from the videos corresponding to each action category to form a query set, and the videos in the query set are used as the videos to be identified (i.e., query videos).

[0063] Construct text descriptions for corresponding action categories: Utilize large language models to... Each action category is described using separate text descriptions, including the start, execution, and end phases. This approach overcomes the limitations of traditional simple labels by employing deep semantic processing of the text data, incorporating a Large Language Model (LLM) for assistance. Specifically, the action category labels of the corresponding video are input into the LLM. A prompt template, "Please break down the following action labels into start, execution, and end phases, and generate corresponding action labels for each phase, ensuring semantic clarity and hierarchical structure," generates three detailed action descriptions with a clear temporal order. For example, these descriptions depict the start, execution, and end phases of the action. For the "shooting" action in a sports video, the descriptions "preparing to hold the ball and bending the knee to jump," "completing the shooting motion in the air," and "the basketball enters the basket and lands" correspond to the start, execution, and end phases, respectively.

[0064] The text description of the action category can also be prepared using a preset text template (e.g., "A video of [CLASS]") before inputting the text encoder. Figure 2 Chinese text description 1, ..., text description These correspond to the text descriptions of the first action category, ..., the second action category, respectively. The text description of each action category is converted into natural language form as input to the text encoder, a technique well known to those skilled in the art.

[0065] S2. Construct an action recognition network model and train it by inputting a support set, a query set, and text descriptions of the corresponding action categories. The action recognition network model includes an image encoder, a text encoder, a cross-modal prototype enhancement module, a text adaptive query alignment module, and a classifier, and is used to perform the following operations:

[0066] S21. Input the query set and support set into the image encoder to extract the query clustering features and support clustering features respectively. Input the text description of each action category into the text encoder to extract the corresponding text features.

[0067] In one embodiment, the image encoder performs the following operations on each video in the query set or support set:

[0068] S211. Encode the video to obtain the corresponding frame-level feature sequence, which contains the visual embedding sequence of several video frames.

[0069] S212. Calculate the similarity between the visual embedding features of any two adjacent video frames in the frame-level feature sequence;

[0070] S213. Merge the visual embedding features of the two adjacent video frames with the highest similarity into a new visual embedding feature. The merging method is to calculate the mean or to use a weighted average.

[0071] S214. Replace the visual embedding features of the two adjacent video frames with the highest similarity with new visual embedding features to update the frame-level feature sequence, and record the length of the updated frame-level feature sequence.

[0072] S215. Return to step S212 until the length of the frame-level feature sequence reaches a preset number. The final frame-level feature sequence is obtained as the clustering feature of the corresponding video. The clustering feature contains If a time-series segment feature is defined, then the clustering features of all videos in the query set are used to form the query clustering features, and the clustering features of all videos in the support set are used to form the support clustering features.

[0073] In one embodiment, the similarity is calculated using cosine similarity.

[0074] Specifically, considering the large amount of redundant information in the video (such as repeated backgrounds or still images), in order to reduce the amount of computation and retain key motion information, the image encoder performs similarity-based temporal clustering on the visual embedding features of the extracted visual frames.

[0075] Specifically, the image encoder encodes each video in the query set or support set to obtain the corresponding frame-level feature sequence. , means as follows:

[0076] =

[0077] in, The current video is the Visual embedding features of each video frame , This represents the total number of video frames in the current video. Then, a bottom-up hierarchical clustering strategy is employed to progressively merge the visual embedding features of adjacent video frames, specifically including the following steps:

[0078] (1) Calculate the similarity between the visual embedding features of any two adjacent video frames in the frame-level feature sequence of the current video. The similarity is calculated using cosine similarity.

[0079] (2) The visual embedding features of the two adjacent video frames with the highest similarity are merged into new visual embedding features. The merging is done by taking the mean or weighted average.

[0080] (3) Replace the visual embedding features of the two adjacent video frames with the highest similarity with new visual embedding features to update the frame-level feature sequence of the current video, and record the length of the updated frame-level feature sequence of the current video.

[0081] (4) Return to step (1) until the length of the current video's frame-level feature sequence reaches the preset number. To obtain the clustering features of the corresponding videos, this embodiment preferably uses... Let each element in the clustering features be a time-series segment feature, that is, the clustering features contain Features of each time segment.

[0082] The above approach introduces a video frame clustering step. Before the cross-modal prototype enhancement module performs cross-modal interaction, video frames in the video are aggregated into compact temporal segments based on similarity, reducing redundant computation. This means the image encoder output contains... The visual embedding features of each video frame are compressed into These features represent key temporal segments, effectively preserving information about crucial stages in the action sequence while reducing background redundancy. The set of clustering features from the support set (videos labeled with action categories) is denoted as the support clustering features. = , , Indicates support for the centralized first Each clustering sub-feature corresponds to a temporal segment feature of the support set. The set of clustering features from the videos in the query set (videos to be identified) is denoted as the query clustering feature. = , Indicates the first in the query set Each cluster sub-feature corresponds to a time-series segment feature of the query set. .

[0083] In one embodiment, the text encoder employs a CLIP model text encoder.

[0084] This embodiment uses a vision-language model (image encoder, text encoder) as the backbone network. The image encoder captures dynamic changes in the video, and uniform sampling is performed on each video segment. Each video frame forms a video segment. This video segment is then input into an image encoder for encoding, yielding visual embedding features containing temporal information. The visual embedding features of each video frame are compressed into This method extracts representative temporal segment features, effectively preserving key stage information during the action process and reducing background redundancy interference. A text encoder is used to extract all... The text descriptions corresponding to each action category are used to generate category-specific text features with rich spatiotemporal semantics, i.e., text features. Alternatively, the image encoder can directly use the image encoder in the existing CLIP model, in which case the image encoder and the text encoder together form the existing CLIP model as the backbone network.

[0085] S22. Input clustering features and all text features into the cross-modal prototype enhancement module to obtain... A semantic enhancement prototype is generated by inputting query clustering features and all text features into a text adaptive query alignment module. Alignment query features, This represents the number of action categories.

[0086] Specifically, for support set features (support clustering features), the Cross-Modal Prototype Enhancement (CMPR) module utilizes fine-grained text features corresponding to the action categories as semantic priors to enhance video features, constructing category prototypes that focus on action-related patterns and suppress background noise. For query set features (query clustering features), the Text Adaptive Query Alignment (TAQA) module is employed to dynamically calibrate the query representation through interaction with the overall text features. This achieves feature reconstruction based on semantic affinity to maximize inter-class separability.

[0087] In one embodiment, clustering features and all text features are input into the cross-modal prototype enhancement module to obtain... There are several semantic enhancement prototypes, each obtained as follows:

[0088] The features of each temporal segment of the current video in the clustering features are concatenated with the corresponding text features in the channel dimension to form the first mixed feature of the corresponding temporal segment features;

[0089] Map the first mixed feature of each time segment feature to the first query vector, and map each time segment feature of the current video to the first key vector and the first value vector respectively;

[0090] Calculate the first attention weight between the first query vector and the first key vector of the corresponding time segment feature. Multiply the first attention weight with the first value vector and then perform weighted aggregation with the first query vector of the corresponding time segment feature to form the first weighted feature.

[0091] The first transformed feature is obtained by using a feedforward neural network to extract features from the first weighted feature. The first transformed feature of the corresponding time segment feature is then residually connected with the first weighted feature and then normalized by layer to obtain the first aggregated feature of the corresponding time segment feature.

[0092] The first aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the semantically enhanced prototype of the corresponding video.

[0093] Existing technologies typically use the raw encoded features output by the image encoder directly for prototype construction, but this often results in background noise unrelated to the action being included in the features. This application introduces explicit textual semantics through a cross-modal prototype enhancement module (CMPR) to calibrate support clustering features, fusing textual features with visual features to obtain an enhanced visual prototype.

[0094] Specifically, such as Figure 2 As shown, for the current video in the supporting clustering features... Features of each time segment Semantic enhancement is performed using the textual features of the corresponding real action category labels to obtain more discriminative cross-modal feature representations, which are then used for prototype construction and classification matching of subsequent support sets. Figure 2 The first query vector of all videos in the support set is denoted as Q1, the first key vector as K1, and the first value vector as V1. The semantic enhancement prototypes of all videos in the support set are denoted as fused support features. The semantic enhancement prototypes for the corresponding videos are obtained using a cross-modal attention mechanism, specifically as follows:

[0095] The first Features of each time segment With the Text features of time-series segments Concatenate along the channel dimension to construct the first hybrid feature, which integrates visual features and textual semantics. Text features of each action category , The number of stages corresponds to the number of stages in the text description of the corresponding action category. For example, if the text description of the corresponding action category is divided into three stages, including the start stage, the process stage, and the end stage, then... Each stage has the characteristics of a temporal segment.

[0096] To achieve deep interaction between visual features and textual semantics, this method employs a cross-modal attention mechanism to obtain a semantically enhanced prototype for the corresponding video, including:

[0097] First, the first mixed feature is mapped to the first query vector. To preserve the spatiotemporal distribution of the current video, the features of each time segment of the current video are mapped to the first key vector and the first value vector, respectively, as shown in the following formula:

[0098]

[0099]

[0100]

[0101] in, Indicates the first Features of a time segment Indicates the first Text features of a time-series segment, Indicates the first The first query vector of the features of each time segment Indicates the first The first key vector of the features of each time segment Indicates the first The first value vector of the features of each time segment , This represents the total number of temporal segment features in the current video. Indicates a splicing operation; A learnable query projection matrix is ​​used to map the first hybrid feature, which combines visual features with textual semantics, to the query space; It is a learnable key projection matrix used to map the features of each temporal segment of the current video to the key space; It is a learnable value projection matrix used to map the features of each temporal segment of the current video to the value space.

[0102] Next, the attention weight between the first query vector and the first key vector is calculated. This attention weight reflects the degree of attention paid to visual features by the text semantics and is applied to the first value vector. Combining residual connections and layer normalization (LN), the first value vector is obtained. The second aggregated feature of each time segment feature The calculation formula is as follows:

[0103]

[0104] In the formula, Representation layer normalization, This represents the softmax function. Indicates transpose. The learnable output projection matrix is ​​used to map the attention-weighted aggregated features back to the dimensional space of the original temporal segment features to generate semantically enhanced temporal segment feature representations. Indicates the first The first key vector of each time segment feature The feature dimension is used as a scaling factor in attention calculation to prevent the dot product result from becoming too large and causing gradient instability. This formula allows us to determine the gradient based on the first feature dimension. The first query vector of features of each time segment The semantic intent of the text in the first The first key vector of each time segment feature Search for relevant visual features and aggregate the corresponding first-order features. The first value vector of the features of each time segment Information.

[0105] Subsequently, the first Aggregated features of time segment features The input is processed through a feedforward neural network (FFN) undergoing a nonlinear transformation, followed by residual connections and layer normalization to obtain the 1st... The final semantic enhancement features of each temporal segment feature :

[0106]

[0107] In the formula, Representation layer normalization, This represents a feedforward neural network.

[0108] Finally, it will support clustering features corresponding to the current video. The refined features of each temporal segment (the final semantic enhancement features) are concatenated along the temporal dimension (temporal dimension concatenation) to obtain the semantic enhancement prototype of the current video. :

[0109]

[0110] in, This indicates concatenation along the temporal dimension. An action category has a semantically enhanced prototype, which embodies the action category.

[0111] In one embodiment, the query clustering features and all text features are input into the text adaptive query alignment module to generate... The alignment query features are obtained as follows:

[0112] Build for the action category of the current video The semantic hypothesis, of which the first... The semantic assumption is that the current video belongs to the [number]th [semantic hypothesis]. Each action category ;

[0113] Under each semantic assumption, a cross-modal attention mechanism is used to obtain the corresponding aligned query features, thereby generating... Alignment query features based on semantic hypotheses are used as the corresponding video. Alignment query features.

[0114] In one embodiment, the corresponding aligned query features are obtained using a cross-modal attention mechanism under each semantic hypothesis. The aligned query features for the k-th semantic hypothesis are obtained as follows:

[0115] The features of each temporal segment of the current video in the query cluster features are concatenated with the text features of the kth action category in the channel dimension to form the second hybrid feature of the corresponding temporal segment features;

[0116] The second mixed feature of each time segment feature is mapped to the second query vector, and the features of each time segment of the current video are mapped to the second key vector and the second value vector respectively.

[0117] Calculate the second attention weight between the second query vector and the second key vector corresponding to the time segment feature. Multiply the second attention weight with the second value vector and then perform weighted aggregation with the second query vector corresponding to the time segment feature to form the second weighted feature.

[0118] The second transformed feature is obtained by using a feedforward neural network to extract features from the second weighted feature. The second transformed feature of the corresponding time segment feature is then residually connected with the second weighted feature and normalized by layer to obtain the second aggregated feature of the corresponding time segment feature.

[0119] The second aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the corresponding video. Alignment query features.

[0120] For the query set, since its action category label is unknown during the action recognition stage, it cannot be semantically guided by directly specifying a single text feature as in the support set. To address this issue, the Text Adaptive Query Alignment (TAQA) module employs a hypothesis-driven dynamic alignment strategy.

[0121] Specifically, for the query clustering features extracted by the image encoder and all Text features for each action category are tested in parallel in this embodiment. There are 1 independent semantic hypothesis. That is, for the 1st... The semantic assumption is that the current video in the query set belongs to the [number]th [video]. The action category, and the first action category. Text features of each action category As a semantic prior .

[0122] The Text Adaptive Query Alignment (TAQA) module is built separately for each action category. An independent semantic hypothesis, such as Figure 2 Semantic hypothesis 1, ..., semantic hypothesis Let Q2 be the second query vector, K2 be the second key vector, and V2 be the second value vector for all videos in the query set. The aligned query features are denoted as fused query features. The corresponding text features, as semantic priors, interact with the current video's clustering features in the query clustering features to generate... The alignment query features are distinct. Specifically, utilizing the same cross-modal attention mechanism as the Cross-Modal Prototype Enhancement Module (CMPR), the calculation is performed on the ... Alignment query features under semantic assumptions The calculation is as follows:

[0123]

[0124] in, This represents the calculation of the cross-modal attention mechanism. The formula represents the... Alignment query features under semantic assumptions The input for calculation is the query set. Clustering features of individual videos and the Text features of each action category In the specific calculation of the cross-modal attention mechanism, the features of each temporal segment of the current video ( The text features of several temporal segment features and the corresponding temporal segment features under the k-th action category are concatenated along the channel dimension to form a second mixed feature of the corresponding temporal segment features. The second mixed feature is used as the second query vector, and the corresponding temporal segment features are used as the second key vector and the second value vector. Cross-modal attention mechanism is then used for computation. Through this process, N different alignment query features will be generated for the same video in the query set, represented as follows: If the semantic assumptions made are correct (i.e., the actual action category of the corresponding video is the first...), then... (For each action category), the text semantics and visual features are highly consistent (i.e., the currently used text description matches the actual content of the video), and the action-related features will be significantly enhanced (semantic resonance occurs); if the semantic assumptions made are wrong, the text semantics and visual features will conflict, and the activation of action-related features will be suppressed.

[0125] S23. Calculate the similarity between all aligned query features and semantically enhanced prototypes of each video in the query set using a classifier, and select the action category corresponding to the semantically enhanced prototype with the highest similarity in the same video as the final action recognition result of the corresponding video in the query set.

[0126] In one embodiment, a dual maximization strategy is adopted to select the action category corresponding to the semantically enhanced prototype with the highest similarity under the same video as the final action recognition result of the corresponding video in the query set.

[0127] In one embodiment, the dual maximization strategy satisfies the following formula:

[0128]

[0129]

[0130] In the formula, This represents the final action recognition result for the corresponding video. Indicates in Find the expression within the parentheses that maximizes its value. , Indicates in Find below The maximum value, Indicates the current video's... The alignment query feature and the first Similarity of semantically enhanced prototypes Indicates the current video's... Alignment query features, Indicates the first A semantically enhanced prototype This indicates the calculation of cosine similarity. This indicates a pooling operation. , .

[0131] The classifier constructs a similarity matrix based on calculations. The size is , where the similarity matrix is ​​the first Line 1 Column elements This represents the current video's [number]. Alignment query features With the A semantic enhancement prototype The similarity. The double maximization strategy is to first perform inner maximization ( For each action category j in the support set that is a candidate, iterate through and query all of the current video. Based on several semantic assumptions, text features that maximize similarity are selected. The goal is to find the alignment query features that best match the candidate action categories, thereby eliminating the interference of irrelevant text semantics on the alignment query features. Then, an outer maximization is performed ( The similarity score is compared among the best matching scores of all action categories selected through the inner maximization filter. The action category with the highest similarity score is the final action recognition result for the current video. The final action recognition result is obtained by calculating the similarity between the aligned query features and the semantically enhanced prototype using a dual maximization strategy.

[0132] S3. Input the support set, the query set to be identified, and the text description of the corresponding action category into the trained action recognition network model to obtain the final action recognition result of the query set to be identified.

[0133] It should be noted that when performing action recognition, the support set may be the same as or different from the videos in the support set used during training, but the action categories must be the same.

[0134] To verify the effectiveness of the few-shot action recognition method based on prototype enhancement and query alignment proposed in this application, experiments were conducted on five publicly available benchmark datasets. These benchmark datasets include: 1) the temporally dominant dataset Something-Something V2 (SSv2 dataset) (including two versions: SSv2-Small and SSv2-Full datasets), characterized by complex temporal dependencies in actions and minimal background contribution; 2) the scene-dominant datasets Kinetics (Kinetics-400 dataset), HMDB51 dataset, and UCF101 dataset, which cover a wide range of human action categories and are primarily used to evaluate the model's generalizability under diverse scenes and appearance features. During training, the hyperparameter settings for each benchmark dataset were as follows: Kinetics-400, HMDB51, and UCF101 datasets: 200 training samples, 10 epochs, learning rate 0.001; SSv2-Small dataset: 1000 training samples, 20 epochs, learning rate 0.001; SSv2-Full dataset: 1000 training samples, 50 epochs, learning rate 0.001. All experiments used the SGD optimizer, selecting 5 action categories. =5, =3.

[0135] Tables 1 and 2 show the accuracy comparison between the method of this application and existing methods (such as TRX model, HCL model, STRM model, HyRSM model, HyRSM++ model, Trokens model, etc.) in the 5-way K-shot setting.

[0136] Table 1

[0137]

[0138] Table 2

[0139]

[0140] Experimental results show that, compared with the best-performing Tokens model in the prior art, the method in this application demonstrates significant advantages on various benchmark datasets, as detailed below:

[0141] Significant improvements on time-dependent datasets: The improvements are particularly evident on the most challenging SSv2 dataset. On the SSv2-Small dataset, the proposed method achieves improvements of 12.0%, 7.2%, and 6.9% in 1-shot, 3-shot, and 5-shot settings, respectively; on the SSv2-Full dataset, improvements are 13.2%, 2.5%, and 1.7%, respectively. This fully demonstrates the effectiveness of the proposed cross-modal refinement and query alignment mechanism in capturing complex temporal actions and eliminating background interference.

[0142] Stable optimizations on scene-driven datasets: On the Kinetics-400 dataset, the proposed method comprehensively outperforms the baseline, with improvements in 1-shot, 3-shot, and 5-shot accuracies of 13.7%, 7.7%, and 6.2%, respectively. On the HMDB51 dataset, the proposed method achieves improvements in 1-shot, 3-shot, and 5-shot accuracies of 16.6%, 10.8%, and 10.0%, respectively; and on the UCF101 dataset, improvements are achieved of 5.1%, 2.3%, and 1.7%, respectively.

[0143] like Figure 3 As shown in Figure (a), the feature distribution extracted by the existing technology (TRX model) exhibits severe class aliasing. Sample points of different colors are intertwined, lacking clear classification boundaries. This indicates that the existing method cannot effectively suppress background noise or capture fine-grained temporal differences when dealing with complex motion videos, resulting in weak feature discriminative power and a tendency to misclassify. Figure 3 In this context, Class0, Class1, Class2, Class3, and Class4 represent the five action categories after classification. In contrast, as shown below... Figure 3 As shown in Figure (b), the feature distribution is significantly improved and the classification boundary is clearer after adopting the method of this application. The intuitive distribution difference strongly demonstrates the effectiveness of the Cross-Modal Prototype Enhancement (CMPR) module and the Text Adaptive Query Alignment (TAQA) module in the method of this application. By introducing semantic priors for feature denoising and dynamic alignment, the method of this application can effectively bring similar samples closer together and push away dissimilar samples, thereby extracting more robust and discriminative action features and significantly improving the action recognition accuracy in small sample scenarios.

[0144] In the table above, the TRX model references Perrett T, Masullo A, Burghardt T, et al. Temporal-relational crosstransformers for few-shot action recognition[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 475-484.; the HyRSM model references Wang X, Zhang S, Qing Z, et al. Hybrid relation guided set matching for few-shot action recognition[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 19948-19957.; and the HyRSM++ model references Wang X, Zhang S, Qing Z, et al. HyRSM++: Hybrid relation guided temporal set matching for few-shot action recognition[J]. Pattern Recognition, 2024, 147: 110110.; STRM model reference Thatipelli A, Narayan S, Khan S, et al. Spatio-temporal relation modeling for few-shot action recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022: 19958-19967.; HCL model reference Zheng S, Chen S, Jin Q. Few-shot action recognition with hierarchicalmatching and contrastive learning[C] / / European conference on computer vision.Cham: Springer Nature Switzerland, 2022: 297-313.; Trokens reference Kumar P, Huang S, Walmer M, et al. Trokens: Semantic-Aware Relational TrajectoryTokens for Few-Shot Action Recognition[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2025: 13544-13556. .

[0145] In summary, the method proposed in this application not only achieves high accuracy in complex temporal action recognition, but also maintains extremely high robustness in general scenarios, demonstrating the powerful generalization ability of the method under different data distributions.

[0146] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0147] The embodiments described above are merely specific and detailed examples of the embodiments described in this application, and should not be construed as limiting the scope of the application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these modifications and improvements all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A few-shot action recognition method based on prototype enhancement and query alignment, characterized in that: Includes the following steps: S1. Construct a support set, a query set, and text descriptions of corresponding action categories based on the motion video dataset; S2. Construct an action recognition network model and train it by inputting a support set, a query set, and text descriptions of the corresponding action categories. The action recognition network model includes an image encoder, a text encoder, a cross-modal prototype enhancement module, a text adaptive query alignment module, and a classifier, and is used to perform the following operations: S21. Input the query set and support set into the image encoder to extract the query clustering features and support clustering features respectively. Input the text description of each action category into the text encoder to extract the corresponding text features. S22, input the support clustering features and all text features into the cross-modal prototype enhancement module to obtain a semantic enhanced prototype, input the query clustering features and all text features into the text adaptive query alignment module to generate an aligned query feature, the number of action categories; S23. Use a classifier to calculate the similarity between all aligned query features and semantically enhanced prototypes of each video in the query set, and select the action category corresponding to the semantically enhanced prototype with the highest similarity under the same video as the final action recognition result of the corresponding video in the query set. S3. Input the support set, the query set to be identified, and the text description of the corresponding action category into the trained action recognition network model to obtain the final action recognition result of the query set to be identified.

2. The few-shot action recognition method based on prototype enhancement and query alignment of claim 1, wherein: The construction of support sets, query sets, and corresponding action category text descriptions based on the action video dataset is as follows: S11, selecting a motion category from the motion video dataset, and extracting a number of videos from the videos corresponding to each motion category support set containing a number of videos and corresponding motion category labels​​ S12、from extracting at least one video from the video corresponding to the action category to form a query set; S13. Utilizing large language models to... Each action category is described in text, including a description of the start phase, the process phase, and the end phase.

3. The few-shot action recognition method based on prototype enhancement and query alignment of claim 1, wherein: The image encoder performs the following operations on each video in the query set or support set: S211. Encode the video to obtain the corresponding frame-level feature sequence, wherein the frame-level feature sequence contains a visual embedding sequence of several video frames; S212. Calculate the similarity between the visual embedding features of any two adjacent video frames in the frame-level feature sequence; S213. Merge the visual embedding features of the two adjacent video frames with the highest similarity into a new visual embedding feature. The merging method is to calculate the mean or to use a weighted average. S214. Replace the visual embedding features of the two adjacent video frames with the highest similarity with new visual embedding features to update the frame-level feature sequence, and record the length of the updated frame-level feature sequence. S215. Return to step S212 until the length of the frame-level feature sequence reaches a preset number. The final frame-level feature sequence is obtained as the clustering feature of the corresponding video, and the clustering feature contains If a time-series segment feature is defined, then the clustering features of all videos in the query set are used to form the query clustering features, and the clustering features of all videos in the support set are used to form the support clustering features.

4. The few-shot action recognition method based on prototype enhancement and query alignment as described in claim 3, characterized in that: The similarity is calculated using cosine similarity.

5. The few-shot action recognition method based on prototype enhancement and query alignment as described in claim 1, characterized in that: The cross-modal prototype enhancement module will support the input of clustering features and all text features. There are several semantic enhancement prototypes, each obtained as follows: The features of each temporal segment of the current video in the clustering features are concatenated with the corresponding text features in the channel dimension to form the first mixed feature of the corresponding temporal segment features; Map the first mixed feature of each time segment feature to the first query vector, and map each time segment feature of the current video to the first key vector and the first value vector respectively; Calculate the first attention weight between the first query vector and the first key vector of the corresponding time segment feature. Multiply the first attention weight with the first value vector and then perform weighted aggregation with the first query vector of the corresponding time segment feature to form the first weighted feature. The first transformed feature is obtained by using a feedforward neural network to extract features from the first weighted feature. The first transformed feature of the corresponding time segment feature is then residually connected with the first weighted feature and then normalized by layer to obtain the first aggregated feature of the corresponding time segment feature. The first aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the semantically enhanced prototype of the corresponding video.

6. The few-shot action recognition method based on prototype enhancement and query alignment of claim 1, wherein: The query clustering feature and the whole text feature are input into a text adaptive query alignment module to generate an aligned query feature, wherein the aligned query feature of each video in the query set is obtained as follows: constructing a semantic hypothesis for an action category of a current video The first semantic hypothesis is that the current video belongs to the first action category The first semantic hypothesis is that the current video belongs to the first action category The first semantic hypothesis is that the current video belongs to the first action category The first semantic hypothesis is that the current video belongs to the first action category Under each semantic assumption, a cross-modal attention mechanism is used to obtain the corresponding aligned query features, thereby generating... Alignment query features based on semantic hypotheses are used as the corresponding video. Alignment query features.

7. The few-shot action recognition method based on prototype enhancement and query alignment as described in claim 6, characterized in that: The alignment query features are obtained by using a cross-modal attention mechanism under each semantic hypothesis. The alignment query features for the k-th semantic hypothesis are obtained as follows: The features of each temporal segment of the current video in the query cluster features are concatenated with the text features of the kth action category in the channel dimension to form the second hybrid feature of the corresponding temporal segment features; The second mixed feature of each time segment feature is mapped to the second query vector, and the features of each time segment of the current video are mapped to the second key vector and the second value vector respectively. Calculate the second attention weight between the second query vector and the second key vector corresponding to the time segment feature. Multiply the second attention weight with the second value vector and then perform weighted aggregation with the second query vector corresponding to the time segment feature to form the second weighted feature. The second transformed feature is obtained by using a feedforward neural network to extract features from the second weighted feature. The second transformed feature of the corresponding time segment feature is then residually connected with the second weighted feature and normalized by layer to obtain the second aggregated feature of the corresponding time segment feature. The second aggregated feature of all temporal segment features of the current video is concatenated along the temporal dimension to obtain the corresponding video. Alignment query features.

8. The few-shot action recognition method based on prototype enhancement and query alignment of claim 1, wherein: The method of selecting the action category corresponding to the semantically enhanced prototype with the highest similarity under the same video as the final action recognition result of the corresponding video in the query set is implemented by a dual maximization strategy.

9. The few-shot action recognition method based on prototype enhancement and query alignment of claim 8, wherein: The dual maximization strategy satisfies the following formula: In the formula, This represents the final action recognition result for the corresponding video. Indicates in Find the expression within the parentheses that maximizes its value. , Indicates in Find below The maximum value, Indicates the current video's... The alignment query feature and the first Similarity of semantically enhanced prototypes Indicates the current video's... Alignment query features, Indicates the first A semantically enhanced prototype This indicates the calculation of cosine similarity. This indicates a pooling operation. , .

10. The few-shot action recognition method based on prototype enhancement and query alignment of claim 1, wherein: The text encoder is a CLIP model text encoder.

Images