An efficient spatio-temporal and motion modeling method for few-shot video action recognition

Abstract

The application provides an efficient spatio-temporal and motion modeling method for few-shot video behavior recognition, aiming to overcome the challenges of insufficient temporal modeling, motion feature fluctuation and insufficient multi-modal fusion of pre-trained models when processing few-shot video tasks. The method is based on the parameter efficient fine-tuning paradigm, and three lightweight trainable modules are integrated on the frozen visual-linguistic model backbone: a cross-frame communication spatio-temporal adapter for enhancing inter-frame dynamic modeling in the backbone network high-order layer; a task-specific motion modulation module for generating task-level motion prior using support set information to align and enhance visual features; and a text-visual interaction channel enhancement module for fusing text semantics through a deep attention and channel re-labeling mechanism to generate a more discriminative multi-modal class prototype. The application realizes excellent recognition performance at a very low computational cost.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and deep learning, specifically to an efficient spatiotemporal and motion modeling method for few-shot video behavior recognition, and more particularly to a few-shot video behavior recognition framework that achieves efficient and scalable modeling by utilizing efficient parameter fine-tuning techniques under limited labeled data conditions. Background Technology

[0002] Video understanding, a core technology in the field of artificial intelligence, faces a major challenge in practical applications: the high cost of labeling high-quality video data. This not only limits the widespread application of deep learning models in real-world scenarios but also led to the emergence of few-shot video action recognition (FSAR) technology, which aims to identify new behavior categories using a small number of labeled samples.

[0003] Current few-shot video action recognition methods mostly follow the metric learning paradigm, which involves constructing category prototypes from a support set (a small number of samples) and then matching query set samples against these prototypes. In recent years, large-scale visual-language pre-trained models have provided valuable prior knowledge for few-shot video action recognition due to their powerful visual-semantic alignment capabilities. However, directly applying contrastive language image pre-trained models to few-shot video action recognition tasks faces three major challenges:

[0004] High computational cost: Although full fine-tuning of the entire contrastive language image pre-trained model (CLIP) yields good performance, it consumes huge computational resources and is not suitable for resource-constrained environments.

[0005] Lack of temporal modeling: Visual encoders pre-trained for contrast language images are mainly designed for processing static images. Their self-attention mechanism focuses on intra-frame feature extraction and lacks explicit modeling of dynamic dependencies between video frames.

[0006] Insufficient use of text priors: Simply using text labels as supervisory signals fails to fully explore the semantics of the text to guide and enhance the discriminative power of the visual prototype, thus limiting the potential of multimodal fusion. Summary of the Invention

[0007] The purpose of this invention is to address the problems existing in the prior art by designing a lightweight fine-tuning framework. This framework effectively enhances the temporal and motion modeling capabilities of the pre-trained model while freezing most of its parameters, and deeply integrates textual semantics to improve the performance of few-shot video action recognition. It provides an efficient and parameter-economical method for few-shot video action recognition. This method is based on a frozen contrastive language image pre-trained model backbone network, cleverly incorporating three collaboratively operating lightweight trainable modules, thereby significantly enhancing the model's capabilities in temporal reasoning, motion discrimination, and multimodal understanding.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: an efficient spatiotemporal and motion modeling method for few-sample video behavior recognition, specifically including the following steps: A language model backbone network with frozen parameters and a text-video dataset are constructed. Multiple lightweight trainable modules are set up within the language model backbone network, and the text-video dataset is divided into training and testing sets. The language model backbone network includes a visual encoder, and a cross-frame communication spatiotemporal adapter is set in the high-level structure of the visual encoder. The cross-frame communication spatiotemporal adapter is used to construct the inter-frame dynamic relationship of the video sequence and enhance the spatial structure. The output port of the visual encoder is equipped with a task-specific motion modulation module. The task-specific motion modulation module generates task-level motion prior knowledge based on the support set video motion information in the current few-sample task, and then performs channel-level modulation on the video features of the support set and query set to obtain modulated support set video features and query set video features. A text-visual interaction channel enhancement module is constructed. For each behavior category, the modulated visual features of the support set samples of that category are concatenated with the semantic features of the corresponding category text labels in the sequence dimension. Cross-modal semantic interaction is established through a multi-head self-attention mechanism, and then channel recalibration is implemented through a squeeze excitation network, thereby generating a multimodal enhancement prototype that integrates visual and text semantics for each category. The temporal alignment metric method is used to calculate the matching distance between the modulated features of the query set video and the multimodal enhanced prototypes of each category, and the query video is classified into the behavior category corresponding to the prototype with the smallest distance.

[0009] Furthermore, the cross-frame communication spatiotemporal adapter includes a time bridge attention module and a dual-path spatiotemporal modeling module; The time bridge attention module is used to establish global cross-frame communication. The specific process includes: extracting classification tokens from the features of each frame in the video sequence, stacking the classification tokens of all frames into a time message matrix in the time dimension, applying a multi-head self-attention mechanism to the time message matrix for cross-frame interaction, and generating an enhanced time message matrix that incorporates the global time context. The dual-path spatiotemporal modeling module is used for local spatiotemporal refinement. Based on the temporal enhancement path, the module integrates the enhanced temporal message matrix into the features of each frame and models short-term local dynamic changes by using channels containing bottleneck structures and three-dimensional convolutions. The spatial enhancement path refines the spatial features of image patch tokens through deep convolutions. After reconstructing the image patch and classification token, it applies a spatial multi-head self-attention mechanism to complete intra-frame context aggregation.

[0010] Furthermore, the specific steps of the task-specific motion modulation module include: Extract forward and backward temporal differences from the features of the support set video as motion cues; The motion cues are processed by a shared encoder, and global average pooling and temporal averaging are performed to extract motion features for each support set sample. The motion features of all support set samples under the same category are averaged to construct a task-level motion embedding. The motion is embedded into two multilayer perceptrons to generate channel-level modulation coefficients, including multiplicative and additive coefficients. The obtained modulation coefficients are applied to the image patch features of the support set and query set videos to perform a channel-level affine transformation operation.

[0011] Furthermore, the specific steps of the text-visual interaction channel enhancement module include: The visual features of the support set and the corresponding text category features are concatenated along the sequence dimension. The concatenated features are input into a multi-head self-attention module to capture global semantic dependencies across modalities; The output of the self-attention module is globally averaged and pooled, and channel attention weights are generated through a squeeze-incentive network. The channel attention weights are multiplied element-wise with the output features of the self-attention module in a broadcast manner to semantically recalibrate the feature channels and ultimately generate an enhanced prototype.

[0012] Furthermore, the temporal bridge attention module extracts output features from each frame. Extract its classification token and pass it through a learnable fully connected layer. Perform a linear transformation:

[0013] Where B is the batch size and C is the feature dimension; Stack the time messages of all frames along the time dimension to construct a time message matrix. Stack all the time messages m_t^{(l)} of all T frames in the time dimension to form a time message matrix M^{(l)}:

[0014] Multi-head self-attention is applied to the time message matrix to achieve cross-frame interaction. The multi-head self-attention mechanism is applied to the time message matrix M^{(l)}, calculating attention along the time dimension, enabling the message at each time step to interact with messages at all other time steps, capturing dependencies spanning longer time intervals.

[0015] Where LN(·) is layer normalization, and MHA(·) is the multi-head self-attention mechanism. It is an updated time message matrix, each element of which incorporates the global time context of the entire video sequence; The updated time message is concatenated back to the original frame features and injected into the global time context. Updated time message splicing back the original frame-by-frame token sequence On the token dimension, an enhanced input is formed:

[0016] This operation injects global temporal context into the features of each frame.

[0017] Furthermore, the time bridge attention module extracts classification tokens from the features of each frame, stacks the classification tokens of all frames into a time message matrix, and applies a multi-head self-attention mechanism to update the time message matrix to achieve cross-frame information communication.

[0018] The beneficial effects of this invention are as follows: Significantly efficient in parameter tuning: This method strictly follows the principle of Parameter Efficient Fine-Tuning (PEFT), requiring only training on a small number of newly introduced module parameters, thereby greatly reducing training costs and resource consumption.

[0019] Enhanced temporal modeling: The cross-frame communication spatiotemporal adapter module effectively compensates for the lack of cross-frame dynamic modeling in the contrastive language image pre-trained model, and improves the model's temporal understanding ability by combining global and local approaches.

[0020] Adaptive motion feature learning: The task-specific motion modulation module uses the prior information of the task itself to modulate the features, enabling the model to learn motion representations that are more relevant and consistent with the specific task, effectively solving the problem of unstable motion features under conditions of few samples.

[0021] Deep multimodal fusion technology: The text-visual interaction channel enhancement module cleverly combines attention mechanisms and channel recalibration strategies to achieve deep guidance of text semantics on visual prototypes, thereby creating multimodal prototypes with more powerful expressive capabilities and significantly improved discriminative power.

[0022] Excellent recognition performance: Significant performance improvements were achieved on multiple few-shot video action recognition benchmark datasets. Experiments were conducted using standard five-class single-sample and five-class five-sample settings for evaluation. On the Something-Something V2 dataset, which features fine-grained actions and strong temporal dependencies, the recognition accuracy reached 67.7% in the five-class single-sample scenario, a 5.6 percentage point improvement compared to the traditional full-scale fine-tuning model's 62.1%; and 74.5% in the five-class five-sample scenario, a 2.4 percentage point improvement compared to the traditional model's 72.1%. On the Kinetics dataset, which contains diverse action categories and complex scene backgrounds, the accuracy reached 94.6% in the five-class single-sample scenario and 96.5% in the five-class five-sample scenario, a 1.1 percentage point improvement compared to the traditional model's 95.4%. On the UCF101 dataset, which covers various scenarios including sports, musical instrument performance, and human-computer interaction, the accuracy rate for single-sample scenarios (five classes) reached 97.0%, on par with traditional models; for five-sample scenarios (five classes), it reached 99.3%, a further improvement over the traditional model's 99.1%, achieving near-perfect recognition performance. On the HMDB51 dataset, which features videos of varying quality and complex backgrounds, the accuracy rate for single-sample scenarios (five classes) reached 81.4%, a 4.3 percentage point improvement over the traditional model's 77.1%; for five-sample scenarios (five classes), it reached 90.2%, a 2.5 percentage point improvement over the traditional model's 87.7%, demonstrating excellent robustness. These results fully validate the effectiveness and generalization ability of the method presented in this invention.

[0023] Broad application prospects: The method of this invention has significant value in practical application scenarios such as intelligent monitoring, human-computer interaction, and behavior prediction. It is particularly suitable for environments where labeled data is scarce and computing resources are limited, providing an efficient and feasible solution for the practical deployment of video understanding technology. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings required in the description of the embodiments or the prior art are briefly introduced below. These drawings only illustrate some embodiments in this application, and those skilled in the art can derive other drawings from these drawings without creative effort.

[0025] Figure 1 The efficient spatiotemporal and motion modeling method for behavior recognition in few-sample videos proposed in this invention adopts a design concept similar to a general technical architecture diagram.

[0026] Figure 2 This is a detailed structural diagram of the Cross-Frame Communication Space-Time Adapter (CFSTA) in this invention.

[0027] Figure 3This is a detailed structural diagram of the Task-Specific Motion Modulation Module (TS-MMM) in this invention.

[0028] Figure 4 This is a structural diagram of the Text Visual Interaction Channel Enhancement Module (TVICE) of the present invention. Detailed Implementation

[0029] To make the technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention: To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0030] like Figure 1 As shown, the E-CFSTM framework proposed in this invention is built on a pre-trained contrastive language image pre-trained visual backbone (such as ViT-B / 16), with most of its parameters frozen. The core innovation lies in the introduction of three trainable lightweight modules: Cross-Frame Communication Spatiotemporal Adapter (CFSTA), Task-Specific Motion Modulation Module (TS-MMM), and Text-Visual Interaction Channel Enhancement Module (TVICE).

[0031] S1: Obtain video data and its text description, construct a text video dataset, preprocess the videos, and divide them into training and test sets.

[0032] S11: Obtain the video dataset and its text descriptions. This embodiment uses the Something-SomethingV2 (SSv2) dataset, which contains approximately 220,000 videos covering 174 action categories. Each video is 2-6 seconds long and is characterized by fine-grained action details and strong temporal dependencies. The video data constitutes a video set, and the corresponding text label descriptions constitute a text set. S12: Divide the text video dataset into a base class set and a new class set according to the few-shot learning paradigm. The base class set is used for model training, and the new class set is used for model testing, ensuring that the categories of the base class set and the new class set do not overlap. S13: Perform segment sampling on the video data, uniformly sampling a fixed number of frame sequences from each video; S14: Use video processing tools to perform frame extraction on the sampled video segments. In this embodiment, each video is sampled with 8 frames, and each frame image is cropped to a resolution of 224×224. S2: Construct a visual language model backbone network with frozen parameters, and set up multiple lightweight trainable modules in the backbone network. like Figure 1As shown, the efficient cross-frame spatiotemporal and motion modeling framework proposed in this invention is built on the pre-trained contrastive language image pre-trained model (CLIP) visual backbone (ViT-B / 16), with most of its parameters frozen. The core innovation of this invention lies in the introduction of three trainable lightweight modules: a cross-frame communication spatiotemporal adapter, a task-specific motion modulation module, and a text-visual interaction channel enhancement module; S21: Load the pre-trained contrastive language image pre-trained model and freeze all the original parameters of the visual encoder and text encoder; S22: The visual encoder uses a Vision Transformer (ViT) with frozen pre-trained weights as its backbone network, and contains 12 transformer coding layers, each consisting of a multi-head self-attention module and a feedforward network. S23: The text encoder is based on a Transformer architecture with frozen pre-trained weights, used to encode text labels of behavior categories into semantic feature vectors; S3: Insert a cross-frame communication spatiotemporal adapter at a higher level of the visual encoder to build inter-frame dynamic relationships in the video sequence and enhance the spatial structure.

[0033] like Figure 2 As shown, a cross-frame communication spatiotemporal adapter is inserted into the high-level structure (layers 10 and 11) of the visual encoder of the contrastive language image pre-trained model to construct inter-frame dynamic relationships and enhance spatial structure of the video sequence. This adapter includes a temporal bridge attention module and a dual-path spatiotemporal modeling module. For the features output from layer l-1... The processing flow of the cross-frame communication spatiotemporal adapter is as follows: S31: Establish global cross-frame communication through the time bridge attention module to build an inter-frame information bridge. The Time Bridge Attention Module is used to establish global cross-frame communication. Its goal is to build an "information bridge" that allows key information from all frames to be collected, interacted with, and distributed back to each frame. S311: Extract classification tokens from the features of each frame and perform a linear transformation. The classification token is considered a condensed representation of the high-level semantics of each frame. The temporal bridge attention module first extracts the output features from each frame. Extract its classification token and pass it through a learnable fully connected layer. Perform a linear transformation:

[0034] Where B is the batch size and C is the feature dimension; S312: Stack the time messages of all frames along the time dimension to construct a time message matrix. Stack all the time messages m_t^{(l)} of all T frames in the time dimension to form a time message matrix M^{(l)}:

[0035] S313: Apply a multi-head self-attention mechanism to the time message matrix to achieve cross-frame interaction. A multi-head self-attention mechanism is applied to the time message matrix M^{(l)} to compute attention along the time dimension, enabling the message at each time step to interact with messages at all other time steps, thereby capturing dependencies spanning longer time intervals:

[0036] Where LN(·) is layer normalization and MHA(·) is multi-head self-attention mechanism. It is an updated time message matrix, each element of which incorporates the global time context of the entire video sequence; S314: Concatenate the updated time message back to the original frame features and inject it into the global time context. Updated time message splicing back the original frame-by-frame token sequence On the token dimension, an enhanced input is formed:

[0037] This operation injects global temporal context into the features of each frame, providing guidance for subsequent modeling; S32: Local spatiotemporal refinement is achieved through a dual-path spatiotemporal modeling module. The dual-path spatiotemporal modeling module includes a temporal enhancement path and a spatial enhancement path, which respectively handle short-term dynamic changes and intra-frame spatial structure; S321: Capturing short-term local dynamic information through time-enhanced paths This path is used to enhance the model's ability to model short-term, local dynamic information in video sequences. It includes bottleneck structures and 3D convolution, and the processing flow is as follows:

[0038]

[0039]

[0040]

[0041]

[0042] in, and These are the weights of the fully connected layer used for channel compression and expansion, forming a bottleneck structure and reducing the reduction rate. =0.25; It is a three-dimensional convolution with a kernel size of 3×1×1, spanning 3 frames in the time dimension, and efficiently capturing local motion patterns; It is the GELU activation function; S322: Refine and enhance the spatial representation of a single frame through a spatial enhancement path. This approach focuses on enhancing the intra-frame discriminative power of features, providing spatial information for identifying categories that look similar but behave differently; S3221: Prepare input features, separate classification tokens and image patch tokens. Features after time-enhanced path processing Separate the classification tokens and image block tokens :

[0043] in, This represents the separated feature sequence, where L is the sequence length. For classification tokens, For image block tokens, This represents a sequence concatenation operation; S3222: Extracting local spatial features through two layers of depthwise convolution. Using two depthwise convolutional layers with a kernel size of 1×3×3, the focus is on extracting local features in the spatial dimension:

[0044]

[0045]

[0046] in, and This represents two layers of depthwise convolution, with the convolution kernel having a temporal dimension of 1 and a spatial dimension of 3×3, and only operating on the spatial region of a single frame; It is the GELU activation function; , , These represent the output of the first convolutional layer, the activated features, and the output of the second convolutional layer, respectively. S3223: Reconstruct spatial features and perform intra-frame context aggregation through a global self-attention mechanism. Refined image patch features are integrated with classification tokens, and intra-frame context aggregation is performed through a global self-attention mechanism.

[0047]

[0048] in, Indicates classifying tokens With enhanced spatial features Reassembly; Residual connection + Ensure stable information flow; MHA(·) represents multi-head self-attention mechanism, and LN(·) represents layer normalization; These are intermediate features after residual connection. The final frame features are output after the spatial enhancement path.

[0049] S4: Set a task-specific motion modulation module at the output of the visual encoder to generate task-level motion priors and modulate video features.

[0050] like Figure 3 As shown, the task-specific motion modulation module is located at the output port of the visual encoder. It is used to generate task-level motion priors based on the video motion information of the support set, and to perform channel-level modulation on the video features of the support set and query set. The specific steps are as follows: S41: Extract forward and backward temporal differences as motion cues from the video features of the support set. For the visual feature tensor after processing by the cross-frame communication spatiotemporal adapter First, it is split into category tokens and image block tokens:

[0051]

[0052] in, Indicates a category token. Represents an image block token; Image block token sequence Reconstructed into frame-by-frame representation Calculate the forward and backward differences:

[0053]

[0054] in, Image patch features representing time step t It is a smoothing operation used to reduce spatial jitter caused by camera shake or the displacement of small objects; S42: Process motion cues through a shared encoder, performing global average pooling and temporal averaging operations to extract motion features. The differential signal is augmented with category-related motion cues using a shared encoder:

[0055]

[0056] in, This represents a shared encoder consisting of lightweight convolutions and non-linear activation functions; Global average pooling is used to aggregate the motion features of each frame in the spatial dimension and then average them in the temporal dimension.

[0057]

[0058] in, and These are C-dimensional vectors, representing the overall forward and backward motion patterns of the video sample, respectively. S43: Average the motion features of all support set samples under the same category to construct the task-level motion embedding. Average the motion embeddings of all K support set samples in the current few-sample task to construct the task-level motion embedding Θ.

[0059] Here, Θ is a C-dimensional vector that encapsulates the common motion characteristics of all known samples in the current task, serving as the task prior. S44: Embed the task-level motion into two multilayer perceptrons to generate channel-level modulation coefficients. Embed the task-level motion into Θ into two independent multilayer perceptrons to generate channel-level multiplicative coefficients λ and additive coefficients μ, respectively.

[0060]

[0061] in, and Each layer contains two fully connected layers and a ReLU activation function; S45: Apply the modulation coefficients to the image patch features of the support set and query set videos, and perform a channel-level affine transformation. The modulation coefficients (λ, μ) are applied to the image patch features of all videos. Perform a channel-level affine transformation:

[0062] Where ⊙ represents element-wise multiplication at the channel level; The modulated image patch features F_p are reconcatenated with the classification token x_{cls} to form motion-modulated video features:

[0063] in, This indicates a concatenation operation along the sequence dimensions. It represents the complete video features after motion modulation, containing 1 classification token and N image block tokens.

[0064] S5: Construct a text-visual interaction channel enhancement module. For each behavior category, this module supports concatenating the modulated visual features of the support set samples of that category with the semantic features of the corresponding category text labels in the sequence dimension. It establishes cross-modal semantic interaction through a multi-head self-attention mechanism, and then performs channel recalibration through a squeeze excitation network, thereby generating a multimodal enhancement prototype that integrates visual and text semantics for each category.

[0065] S51: Average visual features of all support set videos for a category Text features of this category Concatenate along the sequence dimension:

[0066] in, X_{text} represents the visual feature tensor after processing by the aforementioned modules, and X_{text} represents the semantic feature vector obtained by the text encoder after the text label is processed. S52: Input the concatenated features into the multi-head self-attention module to capture global semantic dependencies across modalities. right By applying multi-head self-attention and a feedforward network, deeply fused features are obtained:

[0067]

[0068] in, This represents a multi-head self-attention operation, with three inputs: query, key, and value. Here, we use the self-attention form. For residual connections; This represents a feedforward network, consisting of two fully connected layers and an activation function. Features resulting from enhanced self-attention For the final deep fusion features; S53: Perform global average pooling on the output of the self-attention module, and generate channel attention weights by squeezing the activation network. S531: Perform a squashing operation using global average pooling to generate a global descriptor for each feature channel. Global average pooling is used to aggregate information from the sequence dimension, generating a global descriptor z for each feature channel:

[0069] in, The length of the concatenated sequence (including T frames of visual features and 1 text feature). This represents the feature vector at the i-th position output by the self-attention module; It is a global descriptor that compresses information at the sequence dimension; S532: Performs activation operations through a two-layer fully connected network to generate channel attention weight vectors.

[0070]

[0071] in, These are the weights of the first fully connected layer. These are the weights of the second fully connected layer. , It represents the reduction rate; δ(·) is the ReLU activation function, σ(·) is the Sigmoid activation function; s represents the channel attention weight vector; S54: Multiply the channel attention weights element-wise with the output features of the self-attention module to complete semantic relabeling and generate an enhanced prototype. Semantically relabel the features using the channel attention weights to generate the final multimodal enhanced prototype.

[0072] Here, ⊙ represents channel-level element-wise multiplication, applying the attention weights s to the features at each location in a broadcast manner. This represents the final output multimodal enhanced prototype, which integrates visual and textual semantic information; S6: Classification is performed using a temporal alignment metric, and the network is trained and its performance evaluated. S61: Calculate the matching distance between the query video features and the multimodal augmentation prototypes of each category using the ordered temporal alignment module, and perform dynamic time warping using the ordered temporal alignment module to calculate the alignment distance between the query video features and the support set multimodal augmentation prototypes.

[0073] in, The Ordered Temporal Alignment Module is responsible for dynamic time warping, finding the optimal alignment between two sequences, and calculating the distance. This represents the feature sequence of the support samples over time. Represents the feature sequence of a query sample over a time series. For a given query video, this module will repeat the above steps N times, calculating N alignment distances between it and N category prototypes. Ultimately, the query video is classified into the category corresponding to the prototype with the smallest alignment distance.

[0074] S62: Classify the query video into the corresponding behavior category based on the principle of minimum distance. For a query video, calculate the N alignment distances between it and the N category prototypes. The query video is classified into the category corresponding to the prototype with the smallest alignment distance; S63: Iteratively train the expanded network using the base class set to update the trainable module parameters. The expanded network is iteratively trained using a base class set to update the parameters of the trainable modules. The training process includes forward propagation of network features and backpropagation of errors. In practice, the AdamW optimizer is used, with an initial learning rate of 1e-5, a weight decay coefficient of 0.01, and 20 training iterations. S64: Evaluate recognition performance on the new class set using five-class single-sample and five-class five-sample settings. Evaluation was conducted using five-class single-sample and five-class five-sample settings. On the Something-Something V2 dataset, the recognition accuracy reached 67.7% in the five-class single-sample scenario and 74.5% in the five-class five-sample scenario, representing improvements of 5.6 and 2.4 percentage points respectively compared to the traditional full-scale fine-tuning model. On the Kinetics dataset, the accuracy reached 94.6% in the five-class single-sample scenario and 96.5% in the five-class five-sample scenario. On the UCF101 dataset, the accuracy reached 97.0% in the five-class single-sample scenario and 99.3% in the five-class five-sample scenario. On the HMDB51 dataset, the accuracy reached 81.4% in the five-class single-sample scenario and 90.2% in the five-class five-sample scenario. These results fully validate the synergistic effect of the cross-frame communication spatiotemporal adapter, the task-specific motion modulation module, and the text-visual interaction channel enhancement module in the recognition process, effectively verifying the superior performance of the method of this invention in fine-grained action recognition tasks.

[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient spatiotemporal and motion modeling method for few-sample video action recognition, characterized in that... include: A language model backbone network with frozen parameters and a text-video dataset are constructed. Multiple lightweight trainable modules are set up within the language model backbone network, and the text-video dataset is divided into training and testing sets. The language model backbone network includes a visual encoder, and a cross-frame communication spatiotemporal adapter is set in the high-level structure of the visual encoder. The cross-frame communication spatiotemporal adapter is used to construct the inter-frame dynamic relationship of the video sequence and enhance the spatial structure. The output port of the visual encoder is equipped with a task-specific motion modulation module. The task-specific motion modulation module generates task-level motion prior knowledge based on the support set video motion information in the current few-sample task, and then performs channel-level modulation on the video features of the support set and query set to obtain modulated support set video features and query set video features. A text-visual interaction channel enhancement module is constructed. For each behavior category, the modulated visual features of the support set samples of that category are concatenated with the semantic features of the corresponding category text labels in the sequence dimension. Cross-modal semantic interaction is established through a multi-head self-attention mechanism, and then channel recalibration is implemented through a squeeze excitation network, thereby generating a multimodal enhancement prototype that integrates visual and text semantics for each category. The temporal alignment metric method is used to calculate the matching distance between the modulated features of the query set video and the multimodal enhanced prototypes of each category, and the query video is classified into the behavior category corresponding to the prototype with the smallest distance.

2. The efficient spatiotemporal and motion modeling method for few-sample video behavior recognition according to claim 1, characterized in that: The cross-frame communication spatiotemporal adapter includes a time bridge attention module and a dual-path spatiotemporal modeling module; The time bridge attention module is used to establish global cross-frame communication. The specific process includes: extracting classification tokens from the features of each frame in the video sequence, stacking the classification tokens of all frames into a time message matrix in the time dimension, applying a multi-head self-attention mechanism to the time message matrix for cross-frame interaction, and generating an enhanced time message matrix that incorporates the global time context. The dual-path spatiotemporal modeling module is used for local spatiotemporal refinement. Based on the temporal enhancement path, the module integrates the enhanced temporal message matrix into the features of each frame and models short-term local dynamic changes by using channels containing bottleneck structures and three-dimensional convolutions. The spatial enhancement path refines the spatial features of image patch tokens through deep convolutions. After reconstructing the image patch and classification token, it applies a spatial multi-head self-attention mechanism to complete intra-frame context aggregation.

3. The efficient spatiotemporal and motion modeling method for few-sample video behavior recognition according to claim 1, characterized in that: The specific steps of the task-specific motion modulation module include: Extract forward and backward temporal differences from the features of the support set video as motion cues; The motion cues are processed by a shared encoder, and global average pooling and temporal averaging are performed to extract motion features for each support set sample. The motion features of all support set samples under the same category are averaged to construct a task-level motion embedding. The motion is embedded into two multilayer perceptrons to generate channel-level modulation coefficients, including multiplicative and additive coefficients. The obtained modulation coefficients are applied to the image patch features of the support set and query set videos to perform a channel-level affine transformation operation.

4. The efficient spatiotemporal and motion modeling method for few-sample video behavior recognition according to claim 1, characterized in that: The specific steps of the text visual interaction channel enhancement module include: The visual features of the support set and the corresponding text category features are concatenated along the sequence dimension. The concatenated features are input into a multi-head self-attention module to capture global semantic dependencies across modalities; The output of the self-attention module is globally averaged and pooled, and channel attention weights are generated through a squeeze-incentive network. The channel attention weights are multiplied element-wise with the output features of the self-attention module in a broadcast manner to semantically recalibrate the feature channels and ultimately generate an enhanced prototype.

5. The efficient spatiotemporal and motion modeling method for few-sample video behavior recognition according to claim 2, characterized in that: The temporal bridge attention module extracts output features from each frame. Extract its classification token and pass it through a learnable fully connected layer. Perform a linear transformation: Where B is the batch size and C is the feature dimension; Stack the time messages of all frames along the time dimension to construct a time message matrix. Stack all the time messages m_t^{(l)} of all T frames in the time dimension to form a time message matrix M^{(l)}: Multi-head self-attention is applied to the time message matrix to achieve cross-frame interaction. The multi-head self-attention mechanism is applied to the time message matrix M^{(l)}, calculating attention along the time dimension, enabling the message at each time step to interact with messages at all other time steps, capturing dependencies spanning longer time intervals. Where LN(·) is layer normalization, and MHA(·) is the multi-head self-attention mechanism. It is an updated time message matrix, each element of which incorporates the global time context of the entire video sequence; The updated time message is concatenated back to the original frame features and injected into the global time context. Updated time message splicing back the original frame-by-frame token sequence On the token dimension, an enhanced input is formed: This operation injects global temporal context into the features of each frame.

6. The efficient spatiotemporal and motion modeling method for few-sample video behavior recognition according to claim 5, characterized in that: The time bridge attention module extracts classification tokens from the features of each frame, stacks the classification tokens of all frames into a time message matrix, and applies a multi-head self-attention mechanism to update the time message matrix to achieve cross-frame information communication.

Images