A few-shot action recognition method and device based on local feature enhancement and stage timing decoupling

Abstract

A few-shot action recognition method based on local feature enhancement and stage time sequence decoupling, comprising the following: 1) obtain action video samples of different action categories, preprocess, divide training set, validation set and test set; 2) build a few-shot action recognition model based on local feature enhancement and stage time sequence decoupling, the structure of the few-shot action recognition model based on local feature enhancement and stage time sequence decoupling is: a visual feature branch including a visual feature extraction module and a visual end stage time sequence decoupling module; a text feature branch including a text end stage time sequence decoupling module and a text feature extraction module; two branches converge and enter a multi-modal fusion module and a multi-modal matching module in turn; 3) input the training set into the few-shot action recognition model based on local feature enhancement and stage time sequence decoupling for training, obtain the required trained few-shot action recognition model, and verify the trained model by using the validation set, obtain the final few-shot action recognition model; 4) input the test set into the final few-shot action recognition model, complete the classification prediction of different category video action samples in the test set. The application can guide the model feature response to transfer to the action subject and the interactive object without relying on external pixel-level labeling, weaken the feature interference brought by the complex video background, and provide a clearer spatial visual foundation for few-shot multi-modal alignment.

Description

Technical Field

[0001] This invention relates to the field of computer vision and pattern recognition, and specifically to a method and apparatus for video action recognition using few-sample action recognition technology. Background Technology

[0002] Few-Shot Action Recognition (FSAR) is a primary method for achieving high recognition accuracy with a small dataset, aiming to accurately classify actions using only a few labeled samples. Existing FSAR methods often employ a support-query set metric matching framework, extracting spatiotemporal features and calculating similarity to determine the category. In recent years, to further alleviate the limitations of insufficient training data, researchers have begun to introduce pre-trained visual-language models (such as CLIP) as the backbone network for feature extraction. Because CLIP is pre-trained on large-scale image-text pairs, it possesses strong vocabulary recognition and multimodal alignment capabilities. Introducing it into the FSAR task can significantly improve the model's generalization performance to unseen categories.

[0003] However, many methods struggle to achieve accurate multimodal feature alignment when faced with video sequences exhibiting complex spatiotemporal dynamics. Specifically, the main problems are as follows: First, in the spatial dimension, existing methods are limited by the inherent feature localization defects of pre-trained large visual models, making it difficult to accurately capture the main action subject and the interacting object. Existing CLIP-based models perform poorly in fine-grained feature extraction because directly extracting the last layer output of the CLIP visual encoder results in visual features containing a large amount of background noise, making fine-grained local localization impossible. This often leads to misjudgments in few-sample video matching due to background noise interference. Second, in the temporal dimension, the lack of explicit decoupling of the action's temporal evolution logic leads to the loss of dynamic transition information at action boundaries. This makes it difficult for the model to distinguish easily confused actions with similar visual elements but opposite temporal logic when measuring similarity, and it also masks key dynamic features occurring at action boundaries. Summary of the Invention

[0004] The purpose of this invention is to provide a method and apparatus for few-sample action recognition based on local feature enhancement and stage temporal decoupling, which can improve the model's ability to recognize these few video action samples.

[0005] This invention can be achieved through the following technical solutions: This invention provides a few-shot action recognition method based on local feature enhancement and stage temporal decoupling, comprising the following steps: Step 1: Obtain action video samples of different action categories, preprocess them to obtain the dataset required for few-sample action recognition, and divide the video dataset into training set, validation set and test set. The action categories of the training set, validation set and test set are all different. Step 2: Construct a few-shot action recognition model with local feature enhancement and stage temporal decoupling. The structure of the few-shot action recognition model with local feature enhancement and stage temporal decoupling is as follows: a visual feature branch, including a visual feature extraction module and a visual stage temporal decoupling module; a text feature branch, including a text stage temporal decoupling module and a text feature extraction module; the two branches merge and sequentially enter the multimodal fusion module and the multimodal matching module. The visual feature extraction module is used to extract the visual features fused from the intermediate and final layers of CLIP; the visual-end stage temporal decoupling module is used to extract the atomic action visual sequence index after staged processing; the text-end stage temporal decoupling module is used to extract the staged text sequence output by LLM; the text feature extraction module is used to extract the text features output by CLIP; the multimodal fusion module is used to fuse the features output from the two branches and output the complete action prototype features for classification matching; and the multimodal matching module is used to output the final classification prediction result of the query video. Step 3: Train the few-shot action recognition model with local feature enhancement and stage temporal decoupling using the training set to obtain the desired few-shot action recognition model. Then, validate the trained model using the validation set to obtain the final few-shot action recognition model. Step 4: Input the test set into the final few-shot action recognition model to complete the classification and prediction of different categories of video action samples in the test set.

[0006] Further, step 1 includes: Step 1.1: Collect action videos of several action categories and label each type of action video with an action category tag; Step 1.2: Divide the video sample dataset into a training set, a validation set, and a test set according to a specific ratio. The action categories of the training set, validation set, and test set are all different, so that the model uses non-repeating video samples during training, validation, and testing.

[0007] Further, step 2 includes: Step 2.1: Obtain a continuous video frame sequence of length T, input it into a pre-trained visual encoder, divide each frame into non-overlapping image blocks and map them into one-dimensional vectors, concatenate a category label vector at the beginning of the sequence, and generate a query vector matrix in the feedforward layer of the encoder. Key vector matrix Sum value vector matrix ; Extracting query vectors from the intermediate layers of the visual encoder With key vector Calculate the cross-relevance attention matrix : (1) Performing arithmetic average pooling along the network layers transforms the result into an aggregated intermediate layer attention matrix. : (2) Extract the autocorrelation attention matrix from the output of the last layer of the visual encoder. ,Will and Perform residual weighted fusion to generate a reshaped residual attention matrix. : (3) Using the reshaped residual attention matrix With the value vector matrix of the 12th layer of the visual encoder Multiply, compute the enhanced visual representation, and extract the features corresponding to the category label vectors for each frame as the output set: (4) Step 2.2: Initialize the T time frames in the video into an independent frame index set. Calculate the cosine similarity between adjacent clusters on the time series: (5) By using maximum value retrieval, the index j of the boundary position of the adjacent cluster with the highest similarity value in the sequence is located, and set merging is performed at the index level: (6) (7) Perform arithmetic mean pooling calculations along the time axis to update the mean pooling scale vector for the new cluster. Until the number of remaining clusters equals the preset number of atomic action stages L: (8) Let the size of the overlapping window be... For the Kth stage ( Boundary index set Perform overlapping frame expansion based on the expanded index set. , back to dimension Feature extraction is performed on the original enhanced feature sequence to obtain a tensor with dimension [missing information]. stage entity : (9) (10) Step 2.3: Label the actions at the macro level With the preset structured prompt template Perform concatenation of input to the inference mapping function of the pre-trained large language model In the process, the parsing outputs L atomic text descriptions D with strict temporal dependencies: (11) The generated sequence of atomic text description sentences is input into the text feature encoder. Extract the global semantic embedding vector for each atomic stage to generate L single-dimensional text feature vectors: (12) Step 2.4: Utilize tensor broadcasting mechanism to extract single-sentence text features Copying and expanding, along with the multi-frame visual feature sequence within the corresponding stage. Perform element-wise addition operations to construct the phase query matrix. : (13) By sequentially concatenating the visual sequence sets of all L stages across stages, a global key matrix K and value matrix V are constructed: (14) Calculate the multi-frame feature prototype set after semantic fusion And concatenate along the temporal dimension to generate complete multimodal action prototype features. : (15) (16) Step 2.5: Perform dual-path multimodal matching joint decision-making, specifically including: introducing aligned bidirectional mean Hausdorff distance to calculate the query set complete action prototype. Action prototypes for the i-th action category in the support set Fine-grained set differences: (17) The negative distance scalar is then input into the Softmax activation function to transform it into a video-to-video classification probability distribution: (18) The average visual feature vector of the query set video stages With the k-th atomic text feature of this category Cosine similarity summation aggregation, followed by probability normalization, outputs the video-text matching probability: (19) Introducing weight hyperparameters A weighted geometric average mechanism is used for dual-path fusion to output the final matching decision probability: (20) In the formula, This represents the overall probability score by which the query video q is ultimately classified into the i-th class. Based on this final probability score sequence, the category label corresponding to the highest probability value is retrieved and used as the final action classification result of the few-shot action recognition system, which is then output.

[0008] Furthermore, the process of building a few-shot action recognition model that combines local feature enhancement and stage temporal decoupling also includes: The visual feature extraction module consists of a CLIP visual encoder with frozen parameters for each layer. It extracts the query matrix from the intermediate layers of the encoder, multiplies it with the transposed key matrix, and then performs softmax optimization. The starting layer of the intermediate layers is no less than 50% of the total number of layers in the visual encoder, and the ending layer is no more than 75%. Then, cross-layer average pooling is performed to obtain an attention map, which is weighted and fused with the attention map obtained from the last layer through the same operation. This fusion is then multiplied with the value matrix of the last layer to output the visual features. In the temporal decoupling module at the visual end stage, adjacent visual features of each frame are calculated. The cosine similarity is used to repeatedly merge the maximum values ​​into clusters until the number of clusters reaches the number of stages in the text sequence, and the frame index sequence contained in each stage is output; the text feature extraction module consists of a CLIP text encoder with frozen parameters at each layer, and the last layer outputs text features; in the multimodal fusion module, text features and visual features perform cross-attention calculation, and the action prototype with fused text and visual features is output; the multimodal matching module consists of a metric function calculation unit, a softmax calculation unit, and a matching probability calculation unit, and calculates the final classification prediction probability after weighted averaging.

[0009] Furthermore, the process of building a few-shot action recognition model that combines local feature enhancement and stage temporal decoupling also includes: The input video frame size and dimensions, visual encoder and text encoder input and output sizes and dimensions, number of text sequence stages after LLM processing, sample format required for few-shot action recognition, number of sample classes in N-Way-K-Shot training, and number of samples in each class of K-Shot are all considered.

[0010] A second aspect of the present invention relates to a few-sample action recognition device based on local feature enhancement and stage temporal decoupling, comprising a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a few-sample action recognition method based on local feature enhancement and stage temporal decoupling of the present invention.

[0011] The beneficial effects of adopting the above technical solution are as follows: 1. To address the issues of spatial localization ambiguity and susceptibility to background noise in existing large-scale visual models such as CLIP when extracting video features, this invention proposes a CLIP local feature enhancement mechanism. This mechanism extracts the cross-correlated attention parameters with local localization capabilities from the intermediate layers of the visual encoder and injects them as residuals into the deepest attention layer. This mechanism guides the model's feature response towards the action subject and interactive object without relying on external pixel-level annotations, effectively reducing feature interference from complex video backgrounds and providing a clearer spatial visual foundation for multimodal alignment with few samples.

[0012] 2. To address the shortcomings of traditional methods, such as the lack of decoupling of action temporal logic and the difficulty of providing sufficient fine-grained semantic priors with a single macro-label, this invention proposes a stage-based temporal decoupling and fine-grained cross-modal alignment mechanism. First, through bottom-up frame index clustering and overlapping feature slicing strategies, continuous video is adaptively divided into action stages with specific evolutionary logic. This achieves coherent physical segmentation of actions while effectively preserving high-dimensional spatiotemporal dynamic details at action boundaries. Second, by combining atomic text description sequences generated by a large language model, fine-grained semantics are precisely injected into the decoupled visual stage features. This mechanism enables the model to fully utilize cross-modal alignment of local enhanced features and fine-grained semantic anchors under minimal sample conditions, improving the model's robustness in discriminating complex or easily confused actions. Attached Figure Description

[0013] Figure 1 This is a flowchart of the method of the present invention.

[0014] Figure 2 This is a model structure diagram of the few-sample action recognition model of the present invention, which features local feature enhancement and stage temporal decoupling. Specific Implementation

[0016] The present invention will now be further described with reference to the accompanying drawings: Example 1 This embodiment presents a few-sample action recognition method based on local feature enhancement and stage temporal decoupling, such as... Figure 1 The method includes the following steps: Step 1.1: Collect action videos of several action categories and label each type of action video with an action category tag; Step 1.2: Divide the video sample dataset into a training set, a validation set, and a test set according to a specific ratio. The action categories of the training set, validation set, and test set are all different, so that the model uses non-repeating video samples during training, validation, and testing.

[0017] In this embodiment, the ratio of the training set, validation set, and test set is set to 7:1:2. The division method is based on the proportion of each action category, ensuring that the training set, validation set, and test set do not contain any samples of the same action. This ensures that the datasets used by the model in different experimental stages are all new and unseen, thus ensuring the reliability of the experiment.

[0018] Step 2: Build a few-shot action recognition model with local feature enhancement and stage-time decoupling, such as... Figure 2 The structure of the few-shot action recognition model with local feature enhancement and stage temporal decoupling is as follows: a visual feature branch, including a visual feature extraction module and a visual stage temporal decoupling module; a text feature branch, including a text stage temporal decoupling module and a text feature extraction module; the two branches merge and sequentially enter the multimodal fusion module and the multimodal matching module; The visual feature extraction module consists of a CLIP visual encoder with frozen parameters for each layer. It extracts the query matrix from the intermediate layers of the encoder, multiplies it with the transposed key matrix, and then performs softmax optimization. The starting layer of the intermediate layers is no less than 50% of the total number of layers in the visual encoder, and the ending layer is no more than 75%. Then, cross-layer average pooling is performed to obtain an attention map, which is weighted and fused with the attention map obtained from the last layer through the same operation. This fusion is then multiplied with the value matrix of the last layer to output the visual features. In the temporal decoupling module at the visual end stage, adjacent visual features of each frame are calculated. The cosine similarity is used to repeatedly merge the maximum values ​​into clusters until the number of clusters reaches the number of stages in the text sequence, and the frame index sequence contained in each stage is output; the text feature extraction module consists of a CLIP text encoder with frozen parameters at each layer, and the last layer outputs text features; in the multimodal fusion module, text features and visual features perform cross-attention calculation, and the action prototype with fused text and visual features is output; the multimodal matching module consists of a metric function calculation unit, a softmax calculation unit, and a matching probability calculation unit, and calculates the final classification prediction probability after weighted averaging.

[0019] The visual feature extraction module is used to extract the visual features fused from the intermediate and final layers of CLIP; the visual-end stage temporal decoupling module is used to extract the atomic action visual sequence index after staged processing; the text-end stage temporal decoupling module is used to extract the staged text sequence output by LLM; the text feature extraction module is used to extract the text features output by CLIP; the multimodal fusion module is used to fuse the features output from the two branches and output the complete action prototype features for classification matching; and the multimodal matching module is used to output the final classification prediction result of the query video. Step 2 includes the following specific steps: Step 2.1: Obtain a continuous video frame sequence of length T, input it into a pre-trained visual encoder, divide each frame into non-overlapping image blocks and map them into one-dimensional vectors, concatenate a category label vector at the beginning of the sequence, and generate a query vector matrix in the feedforward layer of the encoder. Key vector matrix Sum value vector matrix ; Extracting query vectors from the intermediate layers of the visual encoder With key vector Calculate the cross-relevance attention matrix .

[0020] Specifically, the video sequence of the action to be identified is acquired. The video is sampled at equal intervals according to a preset temporal sampling strategy to obtain a continuous video frame sequence of length T frames, where T is 8. Spatial normalization and cropping operations are performed on the sampled single-frame images to unify their spatial resolution to 224×224 and the number of channels to 3. At this point, the initial tensor dimension of the input video frame sequence is represented as... .Will The input is fed into a pre-trained visual encoder, where, in the feature embedding layer, each frame of image is divided into segments with a spatial size of [missing information]. Non-overlapping image patches are mapped to feature dimensions using a linear projection operation. A one-dimensional vector is generated and a learnable class token vector is appended to the beginning of the sequence. (Setting...) , After processing, a single frame of video image is converted into a format with a length of [length missing]. The visual marker sequence, the overall video sequence features are transformed into After feature normalization, query vector matrices are generated through linear transformation mapping. Key vector matrix Sum value vector matrix Their tensor dimensions all satisfy ,in The feature dimension of the single-head attention mechanism is 64.

[0021] To obtain spatial relevance information with category specificity and local localization capabilities, a cross-relevance (Query-Key) attention matrix is ​​extracted from 50% to 75% of the total layers of the visual encoder. The query vector from layer i is then used... transpose of key vector The formula for performing matrix multiplication is as follows: (1) In the formula, The elements within this matrix represent the spatial correlation weights between different visual patches within a video frame. To obtain continuous and robust localized spatial response features, an arithmetic mean aggregate pooling operation is performed on the attention matrix extracted within the interval layer, expressed by the following formula: (2) In the formula, N represents the total number of network layers involved in the aggregation calculation, and the intermediate layer attention matrix is ​​the aggregated output. Its tensor dimension is maintained as .

[0022] Subsequently, the attention matrix output from the last layer of the visual encoder is extracted, denoted as... , The output will be aggregated from the intermediate layer attention. and The residual weighted fusion is expressed by the following formula: (3) In the formula, The fusion hyperparameter is set to 0.5. The resulting reshaped residual attention matrix... While inheriting the global semantics of deep networks, it injects a clear spatial local sensitivity distribution, eliminating the spatial response homogenization defect unique to the deepest self-attention mechanism.

[0023] Using the reshaped residual attention matrix Left-multiply the value vector matrix of the 12th layer of the visual encoder Calculate the enhanced visual representation of the current video frame sequence: (4) In the formula, After the tensor dot product operation, features at the corresponding positions of the class token vector for each frame are extracted as a global discriminative representation. The final output enhanced frame-level visual feature set has its dimensionality reduced to [missing value]. That is, it contains 8 frame features, and each feature has a channel dimension of 768.

[0024] Step 2.2: Initialize the visual feature set of the local feature enhancement into an independent set of frame indices; calculate the cosine similarity of features of adjacent frames during the iteration process, and continuously merge the adjacent frames with the highest similarity into clusters until the number of remaining clusters equals the number of target atomic action stages; perform overlapping frame expansion on the final stage boundaries, and return to the original enhanced features for slicing based on the expanded set of frame indices, outputting a set of atomic action visual sequences that retains complete spatiotemporal dynamic details of multiple frames.

[0025] The tensor dimension received from the visual feature extraction module is The visual feature sequence, in the initial state of the clustering algorithm, initializes the 8 time frames in the video into 8 completely independent and non-overlapping initial index clusters, represented as follows: ,in The initial index cluster is assigned an initial feature representation, directly inheriting its corresponding single-frame enhancement features, that is, letting Based on the generated scale vectors, the cosine similarity between adjacent clusters is calculated over time: (5) In the formula, and These represent the 768-dimensional feature representation vectors of the two currently adjacent index clusters. To calculate the cosine similarity score between adjacent clusters in the output.

[0026] Subsequently, the similarity score sequence of all adjacent cluster pairs calculated in the above steps is traversed, and the maximum value retrieval operator is used to lock the boundary position index j corresponding to the adjacent cluster pair with the highest similarity value in the sequence: (6) At the index level, a set merging operation is performed, combining the j-th index set with the (j+1)-th index set into a new time-segmented frame index set by taking the union of the two sets. The calculation formula is expressed as follows: (7) After the index merging is completed, in order to make the newly generated cluster It can be used as a whole parameter in the next round of similarity comparison. Based on all the frame numbers m contained in the new cluster, the corresponding original high-dimensional feature vector is retrieved back. And perform arithmetic mean pooling calculations along the time axis to generate a single representative mean pooling scale vector for updating the new cluster. The calculation formula is expressed as follows: (8) In the formula, This represents the total number of frames contained in the merged new cluster. After compression using this formula, the updated scale vector of the new cluster... The tensor dimension is strictly defined as After completing the above operations, the total number of clusters is decremented by 1, and the above steps will be executed repeatedly until the remaining number of clusters equals the preset number of atomic action stages (3), at which point the iteration stops. At this point, the mean pooling vector used for calculation... They will no longer participate in subsequent calculations, and the output will only retain the set of temporal boundary frame indices obtained from clustering.

[0027] To ensure temporal smoothness of atomic actions at boundaries and to strengthen the temporal connections between clusters at different stages, a frame feature overlap window is set for each adjacent cluster, ensuring that clusters contain the same frame index. Let the overlap window size be... For the Kth stage ( Boundary index set Perform overlapping frame extension: (9) Based on the expanded index set , back to dimension Feature extraction is performed on the original enhanced feature sequence: (10) In the formula, let the number of frames after segmentation be . Then the output is sent to the entity in the next stage. The tensor dimension is It preserves all the multi-frame spatiotemporal dynamic feature sequences.

[0028] Step 2.3: Leveraging the prior knowledge capabilities of the Large Language Model (LLM), a single macroscopic action category label lacking temporal information is decomposed into a text sequence of descriptive sub-steps—preparation, execution, and completion—strictly following the chronological order of action occurrence. Each text sequence clearly represents the action subject, action, and action object at each stage. The generated description follows the natural flow of action: preparation, execution, and completion, thus achieving a more structured text representation with strong temporal information. Subsequently, the text representation is processed by a text encoder to extract individual text feature vectors for each stage, providing fine-grained semantic anchors for cross-modal fusion.

[0029] Obtain macroscopic action category labels to be classified in a few-shot action recognition task. Preset structured action instruction templates Labeling actions at the macro level With the preset structured prompt template The components are then combined and assembled. The prompt word template is mentioned above. It includes generation constraint rules to limit the output format and temporal progression structure of the large language model, requiring it to follow the time progression axis of the physical occurrence of actions, refining macroscopic actions into continuous discrete sub-steps corresponding to the preparation, execution, and completion phases. Subsequently, the composite text prompt stream is input into the inference mapping function of the pre-trained large language model. In the parsing process, three atomic text descriptions D with strict temporal dependencies are output: (11) Where, for any k-th atom text description (in The text statements within it explicitly state the physical subject performing the current sub-action, the specific motion dynamics and posture changes, as well as the objects and scene elements that generate the interaction during the k-th action development stage. This transforms macroscopic action labels that do not contain timeline information into staged, fine-grained text descriptions with strong temporal dependencies.

[0030] When performing multi-stage atomic text feature extraction, the generated sequence of atomic text descriptive phrases D is input into the pre-trained text feature encoder. The text feature encoder network employs a multi-layer Transformer architecture, with its internal channel scale aligned with the feature space of the visual network. At the feedforward input of the text feature encoder, each atomic text description phrase... First, the text tokens are segmented into discrete text token sequences using a word segmentation network. Special end-of-segmentation vectors are then concatenated at designated positions within these sequences, along with text position encoding parameters. After feature embedding, the text token sequences undergo forward feature interaction and semantic depth aggregation within a multi-layer self-attention network. At the top-level output of the text feature encoder network, slice retrieval and feature stripping operations are performed to extract the high-dimensional feature representation of the end-of-segmentation vector at the corresponding position after passing through the entire network's self-attention feature flow. This high-dimensional representation is then used as the global semantic embedding vector for the text description at the k-th atomic stage. , and by Extract the textual feature representation of each atomic stage: (12) In the formula, the tensor dimension of the extracted single text feature is . Its corresponding feature channel dimension of 768 is consistent with the channel dimension of the single-frame enhanced visual features. The system traverses the entire sequence D and finally generates 3 independent single-dimensional text feature vectors. It serves as a structured, fine-grained semantic anchor entity, which is then transmitted as a whole to the subsequent multimodal fusion network.

[0031] Step 2.4: At the atomic stage level, the text feature vectors of each stage are added bit by bit to the multi-frame visual sequence set within the corresponding stage to construct a query matrix. The splicing result of the global multi-stage visual sequence is used as the key and value matrix. Through multi-head cross-attention mechanism and feedforward neural network calculation, the text semantics are injected into the visual features of each frame. Finally, the splicing is used to generate complete action prototype features for classification and matching.

[0032] For the k-th atomic stage, the tensor dimension is... Single-sentence text features With tensor dimension Multi-frame visual sequence set By integrating these technologies and utilizing the tensor broadcasting mechanism at the underlying level of deep learning frameworks, the following methods can be achieved: Copy extension in the frame number dimension Next, align its tensor dimensions with the visual sequence set and convert it to... The expanded text features are compared with the multi-frame visual feature sequences within the corresponding stage. Perform element-wise addition operations to construct the stage query matrix for the multimodal cross-attention mechanism. : (13) In the formula, the completed stage query matrix Through this matrix addition operation, the structured semantics of the text are injected as offsets into the corresponding visual features of each frame.

[0033] Extract the visual sequence set of all three stages from the output of step 2.2, and concatenate the cross-stage feature sequences strictly according to the chronological order of the atomic stages to construct a complete long feature sequence containing global temporal context. This long feature sequence will be used as the key matrix K and value matrix V in the cross-attention network. The formulas for constructing the key and value matrices are expressed as follows: (14) In the formula, let the total number of frames after splicing be... ,but The stage query matrix described in step S41. The global key matrix K and value matrix V are input into a multi-head cross-attention network to calculate a multi-frame feature prototype set after semantic fusion. : (15) In the formula, The prototype sets from each stage are concatenated along the temporal dimension to generate complete multimodal action prototype features for the final few-shot classification metric inference. : (16) In the formula, the global tensor dimension of the complete prototype is... .

[0034] Step 2.5: Based on the generated action prototypes, construct a dual-path multimodal matching mechanism. Path 1 uses the aligned bidirectional mean Hausdorff distance to calculate the fine-grained temporal alignment difference between the query video and the support set videos at multiple atomic stages, outputting the video-video matching probability. Path 2 calculates the cosine similarity between the stage features of the query video and the category atomic text features, outputting the video-text matching probability. Finally, the two probabilities are fused by weighted geometric mean to output the final action category prediction result of the query video.

[0035] Specifically, the video-video matching probability includes obtaining the complete action prototype of the query set videos. Action prototypes for the i-th action category in the support set Introducing an aligned bidirectional mean Hausdorff distance query set prototype. With support set of i-th type prototype The fine-grained set differences. The calculation formula is expressed as: ， (17) In the formula, the search and calculation of distance are strictly restricted to the corresponding k-th stage set, achieving forced alignment of temporal logical evolution. The negative of the calculated distance scalar is then input into the Softmax activation function to transform it into the corresponding video-video classification probability distribution: (18) In the formula, This represents the predicted probability that video q in the query set belongs to the i-th class of the support set under a single-modal video-video matching path.

[0036] The output video-text matching probability includes extracting the visual sequence set of the query set video at the k-th stage of decoupling, performing average pooling calculation on the visual features contained therein along the time frame dimension, and compressing it into a one-dimensional stage average visual feature vector. And calculate the average visual feature vector for this query set stage. With the k-th atomic text feature of this category The cosine similarity is summed and aggregated from the cross-modal similarity scalars calculated in all three stages, and finally, probability normalization is performed on all supporting category dimensions using the Softmax function: (19) Finally, a weighted geometric mean mechanism is used to fuse the video-video and video-text classification probabilities in a dual-path manner, outputting the final matching decision probability that the query set video belongs to the i-th action category: (20) In the formula, This represents the overall probability score by which the query video q is ultimately classified into the i-th class. Based on this final probability score sequence, the category label corresponding to the highest probability value is retrieved and used as the final action classification result of the few-shot action recognition system, which is then output.

[0037] Step 3: Train the few-shot action recognition model with local feature enhancement and stage temporal decoupling using the training set to obtain the desired few-shot action recognition model. Then, validate the trained model using the validation set to obtain the final few-shot action recognition model. In this embodiment, based on the pre-defined N-Way-K-Shot configuration required for few-shot action recognition, the number of sample classes for this training iteration of N-Way is set to 5, and the number of samples within each class of K-Shot is set to 1. Training was performed using 10,000 random samples. The initial learning rate was set to 0.00001. During training, the learning rate was reduced to 0.1 times the initial value after 4,000 random samples, and further reduced to 0.01 times the initial value after 6,000 random samples, allowing the model to converge stably and complete training, thus obtaining the desired few-sample action recognition model. The trained model was then validated using a validation set of 5,000 random samples to obtain the final few-sample action recognition model.

[0038] Step 4: Input the test set into the final few-shot action recognition model to complete the classification and prediction of different categories of video action samples in the test set. In this embodiment, the test set is randomly sampled 5000 times to improve the accuracy of model testing.

[0039] Example 2

[0040] This embodiment relates to a video action recognition device using few-sample action recognition technology, including a memory and one or more processors. The memory stores executable code, and when the one or more processors execute the executable code, they implement a method for video action recognition using few-sample action recognition technology according to Embodiment 1.

[0041] Example 3

[0042] This embodiment relates to a computer-readable storage medium storing a program that, when executed by a processor, implements a method for video action recognition using few-sample action recognition technology as described in Embodiment 1.

[0043] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0044] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A few-sample action recognition method based on local feature enhancement and stage temporal decoupling, characterized in that, The method includes the following steps: Step 1: Obtain action video samples of different action categories, preprocess them to obtain the dataset required for few-sample action recognition, and divide the video dataset into training set, validation set and test set. The action categories of the training set, validation set and test set are all different. Step 2: Construct a few-shot action recognition model with local feature enhancement and stage temporal decoupling. The structure of the few-shot action recognition model with local feature enhancement and stage temporal decoupling is as follows: a visual feature branch, including a visual feature extraction module and a visual stage temporal decoupling module; a text feature branch, including a text stage temporal decoupling module and a text feature extraction module; the two branches merge and sequentially enter the multimodal fusion module and the multimodal matching module. The visual feature extraction module is used to extract the visual features fused from the intermediate and final layers of CLIP; the visual-end stage temporal decoupling module is used to extract the atomic action visual sequence index after staged processing; the text-end stage temporal decoupling module is used to extract the staged text sequence output by LLM; the text feature extraction module is used to extract the text features output by CLIP; the multimodal fusion module is used to fuse the features output from the two branches and output the complete action prototype features for classification matching; and the multimodal matching module is used to output the final classification prediction result of the query video. Step 3: Train the few-shot action recognition model with local feature enhancement and stage temporal decoupling using the training set to obtain the desired few-shot action recognition model. Then, validate the trained model using the validation set to obtain the final few-shot action recognition model. Step 4: Input the test set into the final few-shot action recognition model to complete the classification and prediction of different categories of video action samples in the test set.

2. The few-sample action recognition method based on local feature enhancement and stage temporal decoupling according to claim 1, characterized in that, Step 1 includes: Step 1.1: Collect action videos of several action categories and label each type of action video with an action category tag; Step 1.2: Divide the video sample dataset into a training set, a validation set, and a test set according to a specific ratio. The action categories of the training set, validation set, and test set are all different, so that the model uses non-repeating video samples during training, validation, and testing.

3. The few-sample action recognition method based on local feature enhancement and stage temporal decoupling according to claim 1, characterized in that, The few-shot action recognition model built in step 2, which involves local feature enhancement and stage temporal decoupling, also includes: The visual feature extraction module consists of a CLIP visual encoder with frozen parameters for each layer. It extracts the query matrix from the intermediate layers of the encoder, multiplies it with the transposed key matrix, and then performs softmax optimization. The starting layer of the intermediate layers is no less than 50% of the total number of layers in the visual encoder, and the ending layer is no more than 75%. Then, cross-layer average pooling is performed to obtain an attention map, which is weighted and fused with the attention map obtained from the last layer through the same operation. This fusion is then multiplied with the value matrix of the last layer to output the visual features. In the temporal decoupling module at the visual end stage, adjacent visual features of each frame are calculated. The cosine similarity is used to repeatedly merge the maximum values ​​into clusters until the number of clusters reaches the number of stages in the text sequence, and the frame index sequence contained in each stage is output; the text feature extraction module consists of a CLIP text encoder with frozen parameters at each layer, and the last layer outputs text features; in the multimodal fusion module, text features and visual features perform cross-attention calculation, and the action prototype with fused text and visual features is output; the multimodal matching module consists of a metric function calculation unit, a softmax calculation unit, and a matching probability calculation unit, and calculates the final classification prediction probability after weighted averaging.

4. The few-sample action recognition method based on local feature enhancement and stage temporal decoupling according to claim 3, characterized in that, Step 2 includes: Step 2.1: Obtain a continuous video frame sequence of length T, input it into a pre-trained visual encoder, divide each frame into non-overlapping image blocks and map them into one-dimensional vectors, concatenate a category label vector at the beginning of the sequence, and generate a query vector matrix in the feedforward layer of the encoder. Key vector matrix Sum value vector matrix ; Extracting the query vector from layer s to layer e of the visual encoder With key vector Calculate the cross-relevance attention matrix : （1） Performing arithmetic average pooling along the network layers transforms the result into an aggregated intermediate layer attention matrix. : （2） Extract the autocorrelation attention matrix from the output of the last layer of the visual encoder. ,Will and Perform residual weighted fusion to generate a reshaped residual attention matrix. : （3） Using the reshaped residual attention matrix With the value vector matrix of the 12th layer of the visual encoder Multiply, compute the enhanced visual representation, and extract the features corresponding to the category label vectors for each frame as the output set: （4） Step 2.2: Initialize the T time frames in the video into an independent frame index set. Calculate the cosine similarity between adjacent clusters on the time series: （5） By using maximum value retrieval, the index j of the boundary position of the adjacent cluster with the highest similarity value in the sequence is located, and set merging is performed at the index level: （6） （7） Perform arithmetic mean pooling calculations along the time axis to update the mean pooling scale vector for the new cluster. Until the number of remaining clusters equals the preset number of atomic action stages L: （8） Let the size of the overlapping window be... For the Kth stage ( Boundary index set Perform overlapping frame expansion based on the expanded index set. , back to dimension Feature extraction is performed on the original enhanced feature sequence to obtain a tensor with dimension [missing information]. stage entity : （9） （10） Step 2.3: Label the actions at the macro level With the preset structured prompt template Perform concatenation of input to the inference mapping function of the pre-trained large language model In the process, the parsing outputs L atomic text descriptions D with strict temporal dependencies: （11） The generated sequence of atomic text description sentences is input into the text feature encoder. Extract the global semantic embedding vector for each atomic stage to generate L single-dimensional text feature vectors: （12） Step 2.4: Utilize tensor broadcasting mechanism to extract single-sentence text features Copying and expanding, along with the multi-frame visual feature sequence within the corresponding stage. Perform element-wise addition operations to construct the phase query matrix. : （13） By sequentially concatenating the visual sequence sets of all L stages across stages, a global key matrix K and value matrix V are constructed: （14） Calculate the multi-frame feature prototype set after semantic fusion And concatenate along the temporal dimension to generate complete multimodal action prototype features. : （15） （16） Step 2.5: Perform dual-path multimodal matching joint decision-making, specifically including: introducing aligned bidirectional mean Hausdorff distance to calculate the query set complete action prototype. Action prototypes for the i-th action category in the support set Fine-grained set differences: （17） The negative distance scalar is then input into the Softmax activation function to transform it into a video-to-video classification probability distribution: (18) The average visual feature vector of the query set video stages With the k-th atomic text feature of this category Cosine similarity summation aggregation, followed by probability normalization, outputs the video-text matching probability: （19） Introducing weight hyperparameters A weighted geometric average mechanism is used for dual-path fusion to output the final matching decision probability: （20）。 5. The few-sample action recognition method based on local feature enhancement and stage temporal decoupling according to claim 4, characterized in that, The process of constructing a few-shot action recognition model with local feature enhancement and stage temporal decoupling also includes: the size and dimension of the input video frame of the few-shot action recognition model, the input and output size and dimension of the visual encoder and text encoder, the number of stages of the text sequence after LLM processing, the number of sample categories in the N-Way-K-Shot training of the sample format required for few-shot action recognition, and the number of samples in each category of K-Shot.

6. A few-sample action recognition device based on local feature enhancement and stage temporal decoupling, characterized in that, The device includes a memory and one or more processors, wherein the memory stores executable code, and the one or more processors execute the executable code to implement a few-sample action recognition method based on local feature enhancement and stage temporal decoupling as described in any one of claims 1-5.

Images