Local learnable query enhanced video recognition method

Abstract

The invention relates to the field of image or video recognition, in particular to a local learnable query enhancement video recognition method, which comprises the following steps of: establishing a frame for adding video space-time modeling and migrating to a video task on the basis of an image large model; constructing a video behavior recognition model in the framework; constructing a training mechanism of large model main branch freezing and only new parameter learning, training a video behavior recognition model by using a server, and optimizing an objective function until network convergence to obtain a trained video behavior recognition model; and inputting a video sequence to be recognized into the trained video behavior recognition model to recognize human behaviors. The method has the beneficial effects that a video behavior recognition task can be quickly and efficiently migrated by only adding a small number of learnable parameters on the basis of an image model, human behaviors in a video can be recognized with high precision, and the method has high robustness and effectiveness in the aspect of processing action related details.

Description

Technical Field

[0001] The present invention relates to the field of image or video recognition, and in particular to a video recognition method with local learnable query enhancement. Background Art

[0002] In recent years, video action recognition methods have evolved from CNN-based methods to Transformer-based methods to adapt to changes in mainstream neural networks. Early video Transformers focused on modifying the Transformer architecture for better temporal modeling, such as designing temporal input blocks or temporal attention mechanisms. With the rise of large pre-trained models, methods such as ActionCLIP and XCLIP incorporated CLIP into video action recognition by adding a temporal module. Recent work, such as ILA, introduced implicit learnable alignment for better temporal modeling. Despite their strong performance, these models typically require expensive full-model fine-tuning on video data, limiting wider applications.

[0003] To address this problem, parameter efficient fine-tuning (PEFT) methods have emerged, aiming to adapt the CLIP model to video tasks by keeping most of it frozen and adding only some learnable parameters for efficient training. For example, EVL uses a lightweight Transformer decoder on frozen CLIP to achieve spatiotemporal fusion, and ST-Adapter inserts a temporal adapter before the attention or MLP block. 2 -CLIP further introduces a multimodal adapter and a multi-task decoder to balance supervision and generalization performance. Other methods refine the image-to-video conversion from the perspective of label text, aiming to more clearly represent the action content through language. Nevertheless, most of these methods focus on holistic feature learning, leaving the representation of local action-related regions underexplored, making them vulnerable to irrelevant noise.

[0004] The presence of "attention noise" makes it difficult for the Transformer to accurately retrieve key information, which reduces the efficiency of key feature extraction and hinders performance. This phenomenon is also manifested in video Transformers, where the model captures irrelevant details such as background elements or irrelevant objects. In some cases, the model may even completely ignore the main category of the action, leading to misunderstanding or unrecognizable videos. Although video-level global features remain crucial, noise interference also highlights the importance of action details. Previous methods mainly utilized global features and lacked representation of local, action-related details. Although some methods attempt to detect or represent objects, action-related areas are not always well-defined objects, and how to effectively enhance action-related details remains challenging.

[0005] This paper addresses this problem from two key perspectives: (i) How do we represent action-related regions? One possible solution involves a separate region localization head or detection branch to identify action regions. However, this approach is resource-intensive and heavily relies on labeled region annotations that are not provided by current action recognition datasets. In fact, action details are already embedded in video features, so adaptively extracting these details from global features may help circumvent this problem. (ii) How do we effectively integrate global and action-sensitive region features into a unified model? It is necessary to extract local features from global features while maintaining the harmony of video recognition. Achieving effective fusion without using too many parameters while balancing computational efficiency is a major challenge. This requires careful architecture design, with a focus on controlling the number of learnable parameters to maintain model efficiency. Summary of the invention

[0006] In order to overcome the above-mentioned deficiencies, the present invention aims to provide a video recognition method with local learnable query enhancement, which can achieve fast and efficient migration to video behavior recognition tasks by adding only a small amount of learnable parameters on the basis of the image model.

[0007] The present invention achieves the above-mentioned object through the following scheme: A video recognition method with local learnable query enhancement comprises the following steps:

[0008] S1: Establish a framework based on the image large model and add video spatiotemporal modeling to migrate to video tasks;

[0009] S2: Build a video behavior recognition model within the framework, including:

[0010] Establish a global video feature modeling branch for spatiotemporal modeling and extracting video-level features;

[0011] Establish a feature interaction module for action query and video feature interaction;

[0012] Establish a local action query flow branch with a shared query propagation layer;

[0013] Build a classification head that integrates features from both branches and performs the final classification;

[0014] S3: Build a training mechanism that freezes the main branch of a large model and only learns new parameters. Use the server to train the video behavior recognition model. By optimizing the objective function until the network converges, obtain the local optimal network parameters, and obtain a trained video behavior recognition model.

[0015] S4: Input the video sequence to be identified into the trained video behavior recognition model to identify human behavior.

[0016] Preferably, the specific steps of S1 include:

[0017] S11: Establish a video feature extraction network based on a large image model, first extract frames from a given video, usually uniformly sample even frames, and input them into the video feature extraction network at the same time, extract single-frame features and perform spatiotemporal interactive fusion between different frames, and finally perform average pooling in the temporal dimension to output video features;

[0018] S12: Input the video features into the linear classification layer, obtain and output the classification result, which is the category of the corresponding behavior.

[0019] Preferably, the specific implementation steps of the global video feature modeling branch in S2 include:

[0020] S2-11: Each image-large model Transformer layer consists of L repeated blocks, following the PEFT paradigm. Each block sequentially adds a spatiotemporal adapter, a multi-head self-attention layer, and a fully connected feedforward network. Given the output video embedding of the (l-1)th block: Wherein, N is the number of tokens, T is the number of sampling frames, C is the number of feature channels, and R is a set of real number matrices of shape (T, N, C); S2-12: Next, the lth block performs the following operations. First, the spatiotemporal adaptation ST-AD is performed:

[0021] ST-AD(X l-1 )=X l-1 +Conv3D(X l-1 W dn )W up , where Conv3D represents three-dimensional convolution, W up and W dn They represent the linear layers of dimension increase and dimension reduction in the spatiotemporal adapter respectively; S2-13: Then, the multi-head self-attention MHSA is calculated:

[0022] Where LN represents layer normalization;

[0023] S2-14: Finally, it is fed into the fully connected feedforward network FFN:

[0024] After L layers, the output As a global video feature representation.

[0025] Preferably, the specific implementation steps of the feature interaction module in S2 include:

[0026] S2-21: For the input video feature X, concatenate it with the action query A of the corresponding layer to form the query Q input of the original MHSA of the video Transformer block:

[0027]

[0028] Among them, Q X , Q A denotes the query corresponding to the video feature X and the query corresponding to the action query A of the corresponding layer, respectively. [·] denotes the connection along the dimension of the number of tokens. Meanwhile, the key and value inputs are still the video features:

[0029] Where W Q , W K and W V denote the query, key, and value projection matrices respectively;

[0030] S2-22: Then, the self-attention mechanism is changed to the cross-attention mechanism, and the core operation is expressed as:

[0031] Among them, Softmax represents the Softmax activation function, is the scaling factor, CA represents the cross attention mechanism;

[0032] S2-23: Finally, the output of the video feature is the same as steps S2-13 and S2-14, and the output of the action query can be expressed as:

[0033] Among them, MHCA is a multi-head cross attention mechanism.

[0034] Preferably, the specific implementation steps of the local action query flow branch in S2 include: S2-31: The branch consists of a query propagation (QP) layer and a local spatiotemporal modeling module, for the action query A of the jth layer j and interactive query The QP layer first performs a criss-cross attention mechanism between these inputs:

[0035] Among them, A j As a query, Provide key-value pairs, and the operation is completed frame by frame without any frame interaction;

[0036] S2-32: Then, additional FFN is used to further adjust the features:

[0037] Among them, the QP layer parameters are shared between all L layers, which are regarded as a single QP layer applied iteratively in an autoregressive manner;

[0038] S2-33: Get fully propagated action-related queries Afterwards, a spatiotemporal self-attention mechanism is applied to strengthen the temporal and spatial dependencies between all action queries in all frames. First, the features are transformed from Refactored to S2-34: Then, the following self-attention mechanism is performed along the combined TK dimension:

[0039]

[0040] Preferably, the specific implementation steps of the classification head in S2 include:

[0041] 2-41: Obtain video feature representation X L and local action related queries A ST Then, take X L class tokens, perform average pooling along the time dimension; for A ST , average pooling is performed on the query dimension and the time dimension. Subsequently, these two features are added together to form F, which is then classified through a Linear layer;

[0042] S2-42: Using standard classification cross entropy loss Applied to the joint feature F;

[0043] S2-43: Reconstructing Contrastive Loss Using a Frame-Level Video Feature Inspired by Distillation Loss To specifically guide the study of action query:

[0044]

[0045] Among them, τ is the temperature parameter, A R is through the space-time feature A ST Add a linear layer to adjust the region-specific features, X R Reconstruct the existing target video features, is a negative sample pool consisting of other videos belonging to different categories in the same batch of videos;

[0046] S2-44: The final objective function is defined as:

[0047] Where λ1 and λ2 are the weight factors of these two losses respectively.

[0048] Preferably, the main branch of the large model in S3 freezes the training mechanism of only newly added parameter learning, freezes all parameters of the original image large model, and only learns and trains the parameters of the spatiotemporal adapter in the global video feature modeling branch, the entire local action query stream branch and the linear classification layer of step S12.

[0049] Preferably, the specific implementation steps of S4 include:

[0050] S41: First, extract frames from a given video, extract T frames from each video in a fixed uniform sampling manner, stack them in the batch dimension, and finally form a set of inputs with dimensions BT, C, H, W, where C, H, and W are channels, image height, and width respectively;

[0051] S42: Extract features from the video using the trained video behavior recognition model;

[0052] S43: Input the video features obtained in step S42 into the linear classification layer, and output the category with the highest score as the behavior category corresponding to the entire video.

[0053] Preferably, the even-numbered frames are 8, 16 or 32.

[0054] The beneficial effects of the present invention are as follows: the present invention effectively integrates global and action-sensitive region features into a recognition model, maintains the efficiency of the recognition model by controlling the number of learnable parameters, and allows the learnable query to gradually self-reinforce by designing a local action query flow branch to adapt to the action-related region, capture a full range of action category information, and improve sensitivity to action details. The method can recognize human actions in videos with high accuracy, achieve state-of-the-art performance in multiple action recognition datasets, and verify its effectiveness and robustness in processing action-related details. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic diagram of the steps of the method of the present invention;

[0056] Figure 2 It is a schematic diagram of the overall network framework of the method of the present invention;

[0057] Figure 3 Schematic diagram of an integrated block diagram for realizing effective integration of global and motion-sensitive region features in the present invention; wherein (a) is an integrated block diagram of a global video Transformer and a feature interaction module; (b) is a block diagram of a query propagation layer; (c) is a block diagram of local spatiotemporal modeling;

[0058] Figure 4 is a schematic diagram of a combination of global-local attention according to an embodiment of the present invention;

[0059] Figure 5 is a visualization diagram of the attention score according to an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is further described below in conjunction with specific implementation examples, but the protection scope of the present invention is not limited thereto:

[0061] Example: Figure 1 , Figure 2 As shown, the local learnable query enhanced video recognition method includes the following steps:

[0062] S1: Establish a framework based on the image large model and add video spatiotemporal modeling to migrate to video tasks;

[0063] S2: Build a video behavior recognition model within the framework, including:

[0064] Establish a global video feature modeling branch for spatiotemporal modeling and extracting video-level features;

[0065] Establish a feature interaction module for action query and video feature interaction;

[0066] Establish a local action query flow branch with a shared query propagation layer;

[0067] Build a classification head that integrates features from both branches and performs the final classification;

[0068] S3: Build a training mechanism that freezes the main branch of a large model and only learns new parameters. Use the server to train the video behavior recognition model. By optimizing the objective function until the network converges, obtain the local optimal network parameters, and obtain a trained video behavior recognition model.

[0069] S4: Input the video sequence to be identified into the trained video behavior recognition model to identify human behavior.

[0070] The Top-1 accuracy of this method reached 88.0% on the Kinetics-400 validation set and 72.9% on the Something-something v2 validation set, showing very good recognition effects.

[0071] The following is a more detailed description of each step:

[0072] S1: Establish a framework for migrating video spatiotemporal modeling to video tasks based on a large image model. By migrating a single-frame image model, while retaining the image model, adding a spatiotemporal modeling module at the appropriate location, a video model with performance that exceeds full training or full fine-tuning and is efficiently migrated from the image model is obtained.

[0073] The specific implementation process is as follows:

[0074] S11: Establish a video feature extraction network based on the large image model CLIP. First, extract frames from a given video, usually uniformly sample 8 frames, stack them in the batch dimension, and finally form a set of inputs with dimensions of 8B, 3, H, W, where 3, H, and W correspond to channels, image height, and width, respectively; use the server to randomly generate a set of learnable queries Representing K action-related vectors with C feature channels of 8 frames, this part of the steps allows effective local feature extraction without repeated image input; at the same time, it is input into the video feature extraction network to extract single frame features and perform spatiotemporal interaction fusion between different frames, and finally average pooling is performed in the temporal dimension to output video features;

[0075] S12: Input the video features into the linear classification layer, obtain and output the classification result, which is the category of the corresponding behavior.

[0076] S2: Build a video behavior recognition model within the framework, including:

[0077] (1) Establish a global video feature modeling branch for spatiotemporal modeling and extracting video-level features.

[0078] like Figure 3 As shown in (a), the global spatiotemporal modeling of the video is performed by using the image features of the core units in the Transformer layer. The specific steps are as follows:

[0079] S2-11: Each image-large model Transformer layer consists of L repeated blocks, following the PEFT paradigm. Each block sequentially adds a spatiotemporal adapter (ST-AD), a multi-head self-attention (MHSA) layer, and a fully connected feed-forward network (FFN). Given the output video embedding of the (l-1)th block:

[0080] Wherein, N is the number of tokens, T is the number of sampling frames, C is the number of feature channels, and R is a set of real number matrices of shape (T, N, C); S2-12: Next, the lth block performs the following operations. First, the spatiotemporal adaptation ST-AD is performed:

[0081] ST-AD(X l-1 )=X l-1 +Conv3D(X l-1 W dn )W up ,

[0082] Among them, Conv3D represents three-dimensional convolution, W up and W dn They represent the dimension-increasing and dimension-reducing linear layers in the spatiotemporal adapter respectively;

[0083] S2-13: Then, calculate the multi-head self-attention MHSA:

[0084] Where LN represents layer normalization;

[0085] S2-14: Finally, it is fed into the fully connected feedforward network FFN:

[0086] After L layers, the output As a global video feature representation.

[0087] (2) Establish a feature interaction module for action query and video feature interaction. In order to enable action query to selectively capture features related to local actions in the video, a parameter-free feature interaction module is designed to make full use of the full-scale video features in all layers instead of relying solely on the high-level representation of the last layer. Figure 3 As shown in (a), the video spatiotemporal modeling and feature interaction are performed together with the global video Transformer. The specific steps are as follows: S2-21: For the input video feature X, it is connected with the action query A of the corresponding layer to form the query Q input of the original MHSA of the video Transformer block:

[0088]

[0089] Among them, Q X , Q A denotes the query corresponding to the video feature X and the query corresponding to the action query A of the corresponding layer, respectively. [·] denotes the connection along the dimension of the number of tokens. Meanwhile, the key and value inputs are still the video features:

[0090] Where W Q , W K and W V denote the query, key, and value projection matrices respectively;

[0091] S2-22: Then, the self-attention mechanism is changed to the cross-attention mechanism, and the core operation is expressed as:

[0092] Among them, Softmax represents the Softmax activation function, is the scaling factor, CA represents the cross attention mechanism;

[0093] S2-23: Finally, the output of the video feature is the same as steps S2-13 and S2-14, and the output of the action query can be expressed as:

[0094] Among them, MHCA is a multi-head cross attention mechanism.

[0095] (3) Establish a local action query flow branch with a shared query propagation layer; relying solely on the global video representation will dilute the key spatiotemporal details related to specific actions. This branch is used to dynamically propagate action queries at multiple feature scales in the video representation for self-enhancement, thereby enhancing the model's attention to action-related areas. Figure 3 (b) Figure 3 As shown in (c), this branch allows local action-related queries to effectively capture the spatiotemporal details of all frames. The specific steps are as follows:

[0096] S2-31: Branch consists of a query propagation (QP) layer and a local spatiotemporal modeling module, for the action query A at the jth layer j and interactive query The QP layer first performs a criss-cross attention mechanism between these inputs:

[0097]

[0098] Among them, A j As a query, Provide key-value pairs, and the operation is completed frame by frame without any frame interaction;

[0099] S2-32: Then, additional FFN is used to further adjust the features:

[0100] Among them, the QP layer parameters are shared between all L layers, which are regarded as a single QP layer applied iteratively in an autoregressive manner;

[0101] S2-33: Get fully propagated action-related queries Afterwards, a spatiotemporal self-attention mechanism is applied to strengthen the temporal and spatial dependencies between all action queries in all frames. First, the features are transformed from Refactored to S2-34: Then, the following self-attention mechanism is performed along the combined TK dimension:

[0102]

[0103] (4) Establish a classification head that integrates features from the two branches and performs the final classification. Inspired by the distillation loss, a frame-level video feature reconstruction contrast loss is proposed to specifically guide the learning of action queries. This additional loss focuses on maximizing the alignment of action query features with reconstructed video features, encouraging more accurate local representations. Specifically, it includes the following steps:

[0104] 2-41: Get video feature representation x L and local action related queries A ST Then, take xL class tokens, perform average pooling along the time dimension; for A ST , average pooling is performed on the query dimension and the time dimension. Subsequently, these two features are added together to form F, which is then classified through a Linear layer;

[0105] S2-42: Using standard classification cross entropy loss Applied to the joint feature F;

[0106] S2-43: Reconstructing Contrastive Loss Using a Frame-Level Video Feature Inspired by Distillation Loss To specifically guide the study of action query:

[0107]

[0108] Among them, τ is the temperature parameter, A R is through the space-time feature A ST Add a linear layer to adjust the region-specific features, X R Reconstruct the existing target video features, is a negative sample pool consisting of other videos belonging to different categories in the same batch of videos;

[0109] S2-44: The final objective function is defined as:

[0110] Among them, λ1 and λ2 are weight factors of these two losses respectively, and in a specific embodiment, both are set to 1.

[0111] This embodiment uses a server to perform training of a video behavior recognition model. After the video features are linearly classified, the behavior category in each video is output, with a size of B, N, that is, each video corresponds to a label vector of size N, where N is the number of categories corresponding to the data set.

[0112] Figure 4 The highlighted spots in the global and local branch image sequences represent the attention scores of our method, demonstrating how well it focuses on key action regions.

[0113] S3: Construct a training mechanism that freezes the main branch of the large model and only learns new parameters. Use the server to train the video behavior recognition model, optimize the objective function until the network converges, obtain the local optimal network parameters, and obtain a trained video behavior recognition model. Freeze all parameters of the original image large model, and only learn and train the parameters of the spatiotemporal adapter in the global video feature modeling branch, the entire local action query flow branch, and the linear classification layer of step S12. Use end-to-end training to optimize the objective function objective function λ1and λ2 are 1 until the network converges and obtains the local optimal parameters. The attention score visualization is as follows Figure 5 As shown, the first column represents the attention score of the entire frame, and the next three columns show the distribution of attention scores of the three action queries in the entire frame, indicating how each query focuses on different parts of the video content.

[0114] S4: Input the video sequence to be identified into the trained video behavior recognition model to identify human behavior.

[0115] S41: First, extract frames from a given video. 8 frames are extracted from each video in a fixed uniform sampling manner, and stacked in the batch dimension to form a set of inputs with dimensions of 8B, 3, H, W, where 3, H, and W are channels, image height, and width, respectively.

[0116] S42: Extract features from the video using the trained video behavior recognition model;

[0117] S43: Input the video features obtained in step S42 into the linear classification layer, and output the category with the highest score as the behavior category corresponding to the entire video.

[0118] The method of the present invention can fully integrate global and local features by introducing two branches, namely global video feature modeling and local action query flow, under the drive of the large image model, and relying on the feature interaction module. At the same time, the feature interaction efficiency is improved through the parameter-free attention reuse strategy, and the local query features are effectively integrated with the video features, and finally the most advanced behavior recognition effect can be achieved with a small number of learnable parameters.

[0119] The above description is a specific embodiment of the present invention and the technical principles used. If the changes made according to the concept of the present invention do not exceed the spirit covered by the description and drawings, they should still fall within the scope of protection of the present invention.

Claims

1. A video recognition method with local learnable query enhancement, characterized in that The following steps are involved: S1: Establish a framework based on the image large model and add video spatiotemporal modeling to migrate to video tasks; S2: Build a video behavior recognition model within the framework, including: Establish a global video feature modeling branch for spatiotemporal modeling and extracting video-level features; Establish a feature interaction module for action query and video feature interaction; Establish a local action query flow branch with a shared query propagation layer; Build a classification head that integrates features from both branches and performs the final classification; S3: Build a training mechanism that freezes the main branch of a large model and only learns new parameters. Use the server to train the video behavior recognition model. By optimizing the objective function until the network converges, obtain the local optimal network parameters, and obtain a trained video behavior recognition model. S4: Input the video sequence to be identified into the trained video behavior recognition model to identify human behavior.

2. The video recognition method with local learnable query enhancement according to claim 1, characterized in that: The specific steps of S1 include: S11: Establish a video feature extraction network based on a large image model, first extract frames from a given video, usually uniformly sample even frames, and input them into the video feature extraction network at the same time, extract single-frame features and perform spatiotemporal interactive fusion between different frames, and finally perform average pooling in the temporal dimension to output video features; S12: Input the video features into the linear classification layer, obtain and output the classification result, which is the category of the corresponding behavior.

3. The video recognition method with local learnable query enhancement according to claim 2, characterized in that: The specific implementation steps of the global video feature modeling branch in S2 include: S2-11: Each image-large model Transformer layer consists of L repeated blocks, following the PEFT paradigm. Each block sequentially adds a spatiotemporal adapter, a multi-head self-attention layer, and a fully connected feedforward network. Given the output video embedding of the (l-1)th block: Where N is the number of tokens, T is the number of sampling frames, C is the number of feature channels, and R is a real matrix set with shape (T, N, C); S2-12: Next, the first block performs the following operations. First, it performs spatiotemporal adaptation ST-AD: ST-AD(X l-1 )=X l-1 +Conv3D(X l-1 W dn )W up , where Conv3D represents three-dimensional convolution, W up and W dn They represent the dimension-increasing and dimension-reducing linear layers in the spatiotemporal adapter respectively; S2-13: Then, calculate the multi-head self-attention MHSA: Among them, LN represents layer normalization; S2-14: Finally, it is fed into the fully connected feedforward network FFN: After L layers, the output As a global video feature representation.

4. The video recognition method with local learnable query enhancement according to claim 1, characterized in that: The specific implementation steps of the feature interaction module in S2 include: S2-21: For the input video feature X, concatenate it with the action query A of the corresponding layer to form the query Q input of the original MHSA of the video Transformer block: Among them, Q X , Q A denotes the query corresponding to the video feature X and the query corresponding to the action query A of the corresponding layer, respectively. [·] denotes the connection along the dimension of the number of tokens. Meanwhile, the key and value inputs are still the video features: Among them, W Q ,W K and W V denote the query, key, and value projection matrices respectively; S2-22: Then, the self-attention mechanism is changed to the cross-attention mechanism, and the core operation is expressed as: Among them, Softmax represents the Softmax activation function, is the scaling factor, CA represents the cross attention mechanism; S2-23: Finally, the output of the video feature is the same as steps S2-13 and S2-14, and the output of the action query can be expressed as: Among them, MHCA is a multi-head cross attention mechanism.

5. The video recognition method with local learnable query enhancement according to claim 1, characterized in that: The specific implementation steps of the local action query flow branch in S2 include: S2-31: Branch consists of a query propagation (QP) layer and a local spatiotemporal modeling module, for the action query A at the jth layer j and interactive query The QP layer first performs a criss-cross attention mechanism between these inputs: Among them, A j As a query, Provide key-value pairs, and the operation is completed frame by frame without any frame interaction; S2-32: Then, additional FFN is used to further adjust the features: Among them, the QP layer parameters are shared between all L layers, which are regarded as a single QP layer applied iteratively in an autoregressive manner; S2-33: Get fully propagated action-related queries Afterwards, a spatiotemporal self-attention mechanism is applied to strengthen the temporal and spatial dependencies between all action queries in all frames. First, the features are transformed from Refactored to S2-34: Then, the following self-attention mechanism is performed along the combined TK dimension:

6. The video recognition method with local learnable query enhancement according to claim 1, characterized in that: The specific implementation steps of the classification head in S2 include: S2-41: Obtain video feature representation X L and local action related queries A ST Then, take x L class tokens, perform average pooling along the time dimension; for A ST , average pooling is performed on the query dimension and the time dimension. Subsequently, these two features are added together to form F, which is then classified through a Linear layer; S2-42: Using standard classification cross entropy loss Applied to the joint feature F; S2-43: Reconstructing Contrastive Loss Using a Frame-Level Video Feature Inspired by Distillation Loss To specifically guide the study of action query: Among them, T is the temperature parameter, A R is through the space-time feature A ST Add a linear layer to adjust the region-specific features, X R Reconstruct the existing target video features, is a negative sample pool consisting of other videos belonging to different categories in the same batch of videos; S2-44: The final objective function is defined as: Among them, λ1 and λ2 are the weight factors of these two losses respectively.

7. The video recognition method with local learnable query enhancement according to any one of claims 1 to 6, characterized in that: The main branch of the large model in S3 freezes only the training mechanism of newly added parameter learning, freezes all parameters of the original image large model, and only learns and trains the parameters of the spatiotemporal adapter in the global video feature modeling branch, the entire local action query stream branch, and the linear classification layer of step S12.

8. The video recognition method with local learnable query enhancement according to claim 1, characterized in that: The specific implementation steps of S4 include: S41: First, extract frames from a given video, extract T frames from each video in a fixed uniform sampling manner, stack them in the batch dimension, and finally form a set of inputs with dimensions BT, C, H, W, where C, H, and W are channels, image height, and width respectively; S42: Extract features from the video using the trained video behavior recognition model; S43: Input the video features obtained in step S42 into the linear classification layer, and output the category with the highest score as the behavior category corresponding to the entire video.

9. The video recognition method according to claim 8 is characterized in that The even-numbered frames are 8, 16 or 32 frames.

Images