Zero-sample video action recognition method based on multi-mode prompt learning

Abstract

The invention relates to a zero-sample video action recognition method based on multi-modal prompt learning, and mainly provides a multi-modal prompt fine tuning framework oriented to generalizable video understanding, the framework is based on a visual-language pre-training model CLIP, and the zero-sample video action recognition method based on the multi-modal prompt learning is realized through a soft prompt fine tuning strategy. Effectively migrating cross-modal associated knowledge learned by the CLIP model on massive image-text pairs to a downstream zero sample video action recognition task; the prompt fine tuning strategy comprises LoRA-based fine tuning, cross-frame prompt fine tuning and context equalization fine tuning. According to the invention, the zero sample identification capability and generalization capability of the model can be improved.

Description

Technical Field

[0001] This invention relates to the field of video recognition technology, and more specifically to a zero-shot video action recognition method based on multimodal cue learning. Background Technology

[0002] Zero-shot video action recognition is a video understanding task that aims to identify actions in video samples containing action categories that the model has never seen during training. Traditional video action recognition algorithms, when tested, predict the most likely action category of human activities in an input video from a fixed set of predefined action categories, which cannot meet the requirements of zero-shot video action recognition.

[0003] Current mainstream zero-shot video action recognition technologies are mainly based on fine-tuning of the visual-language pre-trained model CLIP, with the aim of extending the zero-shot performance of the CLIP model to video action recognition tasks. However, the following shortcomings still exist:

[0004] 1) CLIP is a relatively coarse-grained vision-language pre-trained model, which focuses more on extracting global visual information of images; existing related methods (such as VIFI) are directly applied to action recognition tasks, and their performance is poor.

[0005] 2) Video is an ordered collection of images. Methods such as VL prompting and VIFI directly use CLIP to extract features of each frame, which leads to the understanding of each frame being independent of each other and ignores the temporal relationship between the information of frames.

[0006] 3) Existing methods that fine-tune CLIP to adapt to video action recognition tasks usually introduce too many additional learnable modules, which can easily lead to overfitting to downstream training data, causing the model to forget pre-trained knowledge and have relatively limited generalization ability for unseen action classes.

[0007] Therefore, overcoming the above problems is an urgent issue that needs to be addressed by those skilled in the art. Summary of the Invention

[0008] In view of this, in order to at least partially solve the above-mentioned technical problems, the present invention provides a zero-shot video action recognition method based on multimodal cue learning that can be applied to generalized video understanding; through a gentle cue fine-tuning strategy, the cross-modal association knowledge learned by the CLIP model on massive image-text pairs is effectively transferred to the downstream zero-shot video action recognition task.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] First, this invention provides a zero-shot video action recognition method based on multimodal cue learning, the steps of which include:

[0011] Video feature extraction: Classification labels are extracted from the input block embedding sequences of each frame through the Transformer coding layer and aggregated; the aggregated classification labels are concatenated with the corresponding block embedding sequences, and the pre-trained CLIP image encoder is used to generate image features for each frame; average pooling is performed on the image features of each frame to obtain video features;

[0012] Text feature generation: Action category text features are generated using an improved CLIP text encoder; the improved CLIP text encoder includes defining a fixed text template and inserting learnable cue words;

[0013] Video action classification is achieved based on the similarity between video features and text features.

[0014] Preferably, the aggregation process includes layer normalization of classification labels in the block embedding sequence using LayerNorm1 of the CLIP image encoder, and aggregation of cross-frame information using multi-head attention.

[0015] Preferably, the pre-trained CLIP image encoder is used to generate image features for each frame, including:

[0016] After computing spatial self-attention, the classification label appended to the end of the embedding sequence is removed through a slicing operation to make the dimension of the embedding sequence consistent with the original input.

[0017] Preferably, the multi-head self-attention module in the CLIP image encoder and CLIP text encoder is fine-tuned based on LoRA, including: mapping the LoRA to the multi-head self-attention module's mapping matrix W. q W k W v W o In parallel, LoRA is decomposed using low-rank matrices A and B.

[0018] As a preferred option, the fixed text template is "aphoto of aperson doing [class]";

[0019] Add N learnable prompt words X at the beginning and end of a fixed text template to form the structure “XXX[fixed template]XXX”;

[0020] When generating text features for action categories, the word embeddings of the cue word tag X are replaced with learnable vectors.

[0021] Second, this application provides a zero-shot video action recognition system based on multimodal cue learning, including:

[0022] Video feature extraction unit: used to extract and aggregate classification labels from the input block embedding sequences of each frame through the Transformer coding layer; concatenate the aggregated classification labels with the corresponding block embedding sequences, and generate image features for each frame using the pre-trained CLIP image encoder; perform average pooling on the image features of each frame to obtain video features;

[0023] Text feature generation unit: Generates action category text features using an improved CLIP text encoder; the improved CLIP text encoder includes defining a fixed text template and inserting learnable cue words;

[0024] The classification decision unit classifies video actions based on the similarity between video features and text features.

[0025] Third, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, characterized in that, when the processor executes the program, it implements the zero-sample video action recognition method based on multimodal cue learning as described above.

[0026] Fourth, a computer-readable storage medium storing computer instructions thereon, characterized in that, when the instructions are executed by a processor, they implement the zero-sample video action recognition method based on multimodal cue learning as described above.

[0027] The zero-shot video action recognition method based on multimodal cue learning provided by this invention has the following advantages compared with existing technologies:

[0028] 1. By explicitly and gently fusing inter-frame information, the native CLIP image encoder layer is reconstructed into a video encoder layer, improving the model's zero-shot generalization ability.

[0029] 2. Provide heuristic prompt templates and introduce learnable context tokens to enhance task adaptability through end-to-end training; thereby improving the zero-shot recognition performance of the pre-trained model CLIP in unseen action classes.

[0030] 3. By introducing LoRA to perform multimodal joint fine-tuning of the CLIP model, the potential for action recognition inherent in CLIP is effectively activated. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 This is a diagram illustrating the overall framework of the GVU-MPT algorithm of this invention.

[0033] Figure 2 This invention introduces a LoRA-based flowchart for fine-tuning the multi-head self-attention module of CLIP;

[0034] Figure 3 This is a flowchart of the cross-frame information aggregation process based on pre-trained model reuse in this invention.

[0035] Figure 4 This is a flowchart of the generalizable spatiotemporal modeling process based on cross-frame cue fine-tuning of the present invention;

[0036] Figure 5 The flowchart for fine-tuning the balanced context prompts in this invention;

[0037] Figure 6 This is a t-SNE visualization of video features on UCF101 using the CLIP baseline method and GVU-MPT of this invention. Detailed Implementation

[0038] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0040] This invention aims to explore how to effectively extend image-level pre-trained models to video input scenarios based on the visual-language pre-trained model CLIP, thereby improving its zero-shot video action recognition performance. The main directions include exploring suitable fine-tuning methods to activate the latent action-related pre-trained knowledge in the CLIP model and improve its video action recognition performance; finding spatiotemporal modeling methods based on the pre-trained model to achieve better zero-shot video action recognition results; or designing methods to alleviate the overfitting problem of the pre-trained model when fine-tuning for downstream tasks, thereby improving the zero-shot recognition performance of the pre-trained CLIP for unseen action classes.

[0041] To address the aforementioned issues, this application proposes a Generalizable Video Understanding via Multimodal Prompt Tuning (GVU-MPT) framework. Based on the visual-language pre-trained model CLIP, this framework effectively transfers the cross-modal association knowledge learned by the CLIP model from massive image-text pairs to downstream zero-shot video action recognition tasks through a gentle prompting fine-tuning strategy.

[0042] like Figure 1 As shown, the GVU-MPT framework proposed in this invention mainly consists of three parts: multimodal joint fine-tuning based on LoRA (light gray borderless shape selection area), generalizable spatiotemporal modeling based on cross-frame cue fine-tuning in the visual branch (light orange borderless shape selection area), and balanced context cue fine-tuning in the text branch (light blue borderless shape selection area).

[0043] These three core mechanisms minimize the impact on the gradient flow of the frozen CLIP for all extensions, and additionally incorporate a shared parameter reuse design to mitigate overfitting in the introduced LoRA, resulting in a more ingenious and gentle overall approach. In terms of performance, this invention improves the accuracy of visible action recognition while maintaining the model's generalization ability to unseen classes.

[0044] The following is a description through specific embodiments.

[0045] Example 1: Multimodal joint fine-tuning based on LoRA;

[0046] To enable CLIP to better focus on action-related cues and contextual relationships, thus adapting it to video action recognition tasks, this invention introduces LoRA fine-tuning technology for large language models. LoRA technology is used to fine-tune the multi-head self-attention modules of CLIP's visual encoder and text encoder, improving the attention distribution of the original multi-head attention layers. Its structure is as follows: Figure 2 As shown.

[0047] Specifically, in each Transformer encoding layer of the native CLIP visual encoder and text encoder, the mapping matrix W between LoRA and the multi-head self-attention module is... q W k W v W o Parallel decomposition is performed using two low-rank matrices A and B. For each pre-trained frozen projection matrix W∈R... d×d Decompositional incremental update is used:

[0048] h = Wz + BAz = (W + BA)z

[0049] Where A∈R r×d Let B be a dimension-reduced matrix, and B ∈ R. r×d Let z be the up-dimensional matrix, z be the input embedding, and h be the output after LoRA adaptation and fine-tuning.

[0050] The calculation process can be represented as follows:

[0051] Q=zW′ q =z(W q +A q B q )

[0052] K=zW′ k =z(W k +A k B k )

[0053] V=zW′ v =z(W v +A v B v )

[0054] z attn =Attention(Q,K,V)W′ o =Attention(Q,K,V)(W o +A o B o )

[0055] Among them, W q W k W v W o Let A be the pre-trained weight matrix. q A k A v A o and B q B k B v B o The low-rank adaptation matrix introduced for LoRA.

[0056] During fine-tuning training, all pre-trained weights are frozen, and only low-rank matrices A and B are updated. Matrix A is initialized using Gaussian initialization, and matrix B is initialized with zero. This ensures that the incremental weights ΔW = BA = 0 at the beginning of fine-tuning training, preventing the introduction of additional noise into the pre-trained model.

[0057] Compared to other fine-tuning methods, the incremental fine-tuning update mechanism allows for a smoother training process. The number of learnable parameters is significantly reduced compared to full fine-tuning, while also not introducing additional inference latency during inference.

[0058] Overall, traditional fine-tuning methods may destroy the general representation capabilities of CLIP pre-training. This invention, through the LoRA low-rank update mechanism, can target and optimize key modules to make them more suitable for video tasks.

[0059] In CLIP's visual encoder, LoRA-adapted multi-head attention layers are fine-tuned to make the model focus more on key action cues in the video, rather than static backgrounds. Furthermore, while traditional methods typically adjust the visual or text branches separately, this invention simultaneously fine-tunes CLIP's image-text dual encoder using LoRA, ensuring alignment between video features and action category text features and improving cross-modal matching accuracy.

[0060] Example 2: Generalizable spatiotemporal modeling based on cross-frame cue fine-tuning in the visual branch;

[0061] In this embodiment, the modeling steps include:

[0062] The classification labels are extracted from the input block embedding sequences of each frame by the Transformer coding layer and aggregated; the aggregated classification labels are concatenated with the corresponding block embedding sequences, and the pre-trained CLIP image encoder is used to generate the image features of each frame; the image features of each frame are averaged and pooled to obtain the video features.

[0063] In this embodiment, the above process mainly includes two parts: cross-frame information aggregation based on pre-trained model reuse and fine-tuning based on video context guidance.

[0064] 2.1 Cross-frame information aggregation based on pre-trained model reuse

[0065] like Figure 1 For any video V, the frame sequence V = {I1, I2, ..., I} is obtained by sampling from the video. 32}. Any frame I t ∈R H×W×3 First, the data is divided into M fixed-size patches, each 16×16 pixels. Each patch is then linearly projected into a high-dimensional embedding vector. Finally, all the patch embeddings are arranged in a left-to-right, top-to-bottom order to form a patch embedding sequence E0∈R. M×768 Specifically, an additional learnable classification token ([class]token) c0∈R is appended to the beginning of the input sequence. 1×768 ,Right now:

[0066] z0 = [c0, E0]

[0067] Where z0 is the block embedding input sequence for the subsequent coding layer.

[0068] like Figure 3This application embeds the classification tags embedded in the input sequence of each Transformer coding layer as frame-level representations. Classification tags for each frame are extracted from the input sequence of all video frames to form a classification tag sequence. Then, the multi-head self-attention module, finely tuned with LoRA adaptation as described earlier, is reused to facilitate information exchange between classification tags.

[0069] It is important to note that the multi-head self-attention module reused here loads the pre-trained weights of the corresponding multi-head self-attention layer in the native CLIP image encoder, and keeps them frozen during subsequent training.

[0070] In one embodiment, taking the (i+1)th Transformer coding layer in the video encoder as an example, the aggregation process is as follows:

[0071] First, extract the classification tags from the embedded sequences of each frame of the video:

[0072]

[0073] Among them, C i This represents a sequence consisting of classification labels for each frame, where T is the number of video frames and 768 is the embedding dimension.

[0074] Subsequently, LayerNorm1 of the corresponding native CLIP image encoder is reused for layer normalization:

[0075]

[0076] Next, the normalized classification label sequence is fed into the multi-head attention module with LoRA adaptation fine-tuning to aggregate cross-frame information:

[0077]

[0078] Among them, W q W k W v Let A be the pre-trained weight matrix. q A k A v and B q B k B v The low-rank adaptation matrix introduced for LoRA.

[0079] Then calculate the attention score:

[0080]

[0081] Finally, the updated classification label sequence is calculated using the output mapping matrix of the LoRA adaptation:

[0082] C′i =Attention(Q,K,V)W′ o =Attention(Q,K,V)(W o +A o B o )

[0083] Among them, W o Let A be the pre-trained output mapping matrix. o and B o The low-rank adaptation matrix introduced for LoRA. The encoded output classification label sequence C′. i ∈R T×768 .

[0084] Through the above calculation process, cross-frame information aggregation based on pre-trained models can be achieved, promoting information exchange between classification labels.

[0085] 2.2 Fine-tuning of prompts based on video context guidance

[0086] After completing the cross-frame information aggregation based on pre-trained model reuse, this invention designs a video context-guided prompt fine-tuning mechanism. It utilizes classification tags that integrate video context information to assist subsequent spatial modeling, and together with the cross-frame information aggregation mechanism described above, forms a complete generalizable spatiotemporal modeling mechanism based on cross-frame prompt fine-tuning.

[0087] The specific structure is shown in Figure 4. In each sublayer of the video encoder proposed in this invention,

[0088] The classification tag sequence C, which aggregates contextual information from the previous text, is used as an example. ' i The data is then dispersed again and concatenated with the block embedding sequences corresponding to each classification label to obtain a new frame embedding sequence input z. ' i,t :

[0089]

[0090] Among them, z i,t z represents the embedded sequence input corresponding to the t-th frame in the (i+1)-th layer of the video encoder. i,t ∈R (M+1)×768 ;C′ i,t C′ represents the classification label of the aggregation context corresponding to frame t. i,t ∈R 1×768 .

[0091] Subsequently, the corresponding Transformer encoding layer, fine-tuned with LoRA adaptation as described above, is used to perform context-guided spatial self-attention modeling. This computational process can be represented as follows:

[0092]

[0093] Among them, W q W k W v Let A be the pre-trained weight matrix. q A k A v and B q B k B v The low-rank adaptation matrix introduced for LoRA.

[0094] Then perform spatial self-attention operations:

[0095]

[0096] Furthermore, from slicing operations Remove the video context classification tag C that was previously appended to the end of the embedded sequence. ' i,t :

[0097]

[0098] Where M = 196, [0:M+1] represents the slicing operation, i.e., only keeping The first M+1 rows.

[0099] This ensures that the dimensions of the embedded sequence are consistent with the original input, while preserving the results of spatial attention modeling.

[0100] After residual connection, layer normalization is performed again:

[0101]

[0102] It is the final output embedding of the t-th frame at the i-th video coding layer. It is also the input to the next video coding layer.

[0103] Then, the output of the last video coding layer Classification tags in The extracted features are normalized and projected into a unified visual language feature space as the image features of the t-th frame.

[0104]

[0105] Where Vision.proj represents the MLP at the end of the native CLIP image encoder,

[0106] Finally, average pooling is performed on the image features of each frame to obtain the video features:

[0107]

[0108] exist Figure 4 In the illustrated framework, the first LoRA-tuned multi-head attention module (MSA) aims to aggregate inter-frame information of the image, while the second LoRA-tuned MSA focuses on aggregating intra-frame information. These two modules are highly correlated in their task objectives and are close to each other in the data flow path. This invention designs a novel constraint strategy: allowing the two LoRA-tuned MSA modules (including LoRA parameter matrices A and B) in the same video encoder layer to share weights, specifically implemented by using the same MSA layer. This design has two advantages: firstly, by sharing weights, the number of parameters introduced by LoRA tuning can be reduced, thus alleviating overfitting; secondly, during training, the module is constrained by both the inter-frame and intra-frame information aggregation requirements of the image. This constraint forces the model to learn the refined core knowledge shared between these two requirements, thereby achieving stronger generalization ability.

[0109] Example 3: Fine-tuning of balanced contextual hints in text branches;

[0110] like Figure 5 As shown, this application first creates a fixed prompt template "a photo of a person doing [class]" for each action category (such as running, jumping, combing hair).

[0111] Compared to the commonly used "a video of [class]" in zero-shot video understanding work, this invention argues that CLIP is trained on a massive amount of image-text pairs, and its training data does not contain videos. The CLIP pre-trained model stores more static knowledge about "photo," so to closely match the existing knowledge of the pre-trained model and provide a good contextual cue template, this invention uses "photo" instead of "video." Furthermore, unlike ordinary zero-shot image classification tasks, the subject of video action recognition is always human, so the contextual cue in the text can be limited to "person."

[0112] In addition, this invention designs a learnable text prompt template based on the manual prompt template. This text prompt template consists of 2N learnable text prompt words, which are symmetrically distributed: N at the beginning of the sentence and the other N before the punctuation at the end of the sentence. Here, N is a hyperparameter, which is assumed to be N=3 for simplicity in the following description.

[0113] Overall, GVU-MPT constructs a new text input for the text encoder branch: “XXX aphotoofaperson doing[class]XXX.”, where X represents a learnable cue word. Formally, given an action category, such as “jogging,” the specific text input is “XXX aphoto ofa person doing jogging XXX.”. CLIP's text encoder first segments the text input into tokens, and then tokenizes each token into a numeric ID. Furthermore, to facilitate batch processing, [SOT] (sentence start token), [EOT] (sentence end token), and [Pad] (padding token) are added to the beginning and end of each text token sequence, respectively, resulting in a fixed sequence length of 77. These obtained token sequences are then mapped into 512-dimensional semantic embedding vectors by the word embedding layer.

[0114]

[0115] Where w0 represents the text embedding obtained through the word embedding layer. Let X represent the word embedding vector corresponding to the i-th lexical unit in the sentence. Then, the word embedding of the prompt word tag X is replaced with a learnable vector to achieve the effect of learnable prompt words.

[0116]

[0117] in, These are the word embedding vectors of [SOT] (Sentence Start Marker), v0, v1, v2, and v 11 ,v 12 ,v 13 These correspond to the learnable prompt words at the beginning and end of the sentence in the text input "XXX aphoto of a person doing jogging XXX." This corresponds to [EOT] (sentence end marker) and [Pad] (pad marker).

[0118] These learnable cue words are optimized end-to-end, aiming to learn a general and widely applicable cue template, rather than being specific to a particular action category. Up to this point, the balanced contextual cue fine-tuning method proposed in this invention has only modified the input to the CLIP text branch without altering the structure of any pre-trained model.

[0119] The text embedding w'0 is then fed into 12 text encoding layers T enhanced with LoRA adaptation. i-LoRA The output embedding of the i-th sub-layer can be represented as:

[0120] w i =T i-LoRA (w i-1 )

[0121] Where i = 1, 2, ..., k, w i ∈R 77×512 Since the computation process for each LoRA-adapted enhanced text encoding layer is consistent with the previous steps, it will not be described again. The subsequent process is completely identical to the native CLIP, taking the output sequence w of the last sub-layer of the text encoder. 12 The output at the location of [EOT] is extracted and mapped to obtain the final text features. This process can be represented as:

[0122]

[0123] Among them, t f ∈R 1×512 Text.proj represents the MLP at the end of the native CLIP text encoder.

[0124] Example 4: Video action classification based on the similarity between video features and text features;

[0125] GVU-MPT ultimately extracts a text feature for each category of text. Extracting video features v from video input using a reconstructed video encoder f The cosine similarity between video features and S visible category text features is calculated to obtain the cosine similarity score between the video and the S visible categories. Then, the cosine similarity scores of the visible categories are normalized to calculate the probability distribution of the input video belonging to the visible action category.

[0126]

[0127] Where p(y) i |V) indicates that the video input V belongs to category y. i The probability. The video input is ultimately identified as the action category with the highest classification probability.

[0128] During training, CLIP's text encoder and image encoder are frozen, and their parameters are not updated. Only the LoRA and learnable cue words in the balanced contextual cue fine-tuning method are trained. These parameters are optimized by minimizing the cross-entropy loss between the model's predictions and the training data labels. The formula illustrates the calculation process of the cross-entropy loss:

[0129]

[0130] Compared with existing technologies, the model trained by the framework proposed in this invention has higher performance and better generalization ability on zero-shot image classification tasks, specifically including:

[0131] First, a multimodal fine-tuning based on LoRA is proposed. Through the LoRA low-rank update mechanism, key attention modules are optimized in a targeted manner, making the model focus more on key action cues in the video rather than static noise background, thus adapting CLIP to video tasks. Furthermore, while existing methods typically adjust the visual or text branches separately, this invention simultaneously fine-tunes the image-text dual encoder of CLIP using LoRA, ensuring alignment between video features and action category text features, and improving cross-modal matching accuracy.

[0132] Secondly, a prompt-based generalizable spatiotemporal modeling method is proposed, which integrates inter-frame information in an explicit and gentle manner, reconstructs the native CLIP image encoder layer into a video encoder layer, fills the gap in temporal modeling of the CLIP image encoder, and improves the zero-sample generalization ability of the model.

[0133] Then, a balanced contextual cue fine-tuning strategy is proposed: First, a heuristic, general basic cue template is customized for the zero-shot video action recognition task to ensure the stable representation of pre-trained knowledge; then, learnable context tokens are introduced based on the heuristic cue template, and task adaptability is enhanced through end-to-end training. This balanced design preserves the generality of pre-trained knowledge, while effectively preventing overfitting through implicit constraints on the learning space, thereby improving CLIP's zero-shot recognition performance for unseen action classes with a balanced cue template.

[0134] To facilitate the transformation and application of the recognition method in this application, this application further provides a zero-shot video action recognition system based on multimodal cue learning, including:

[0135] Video feature extraction unit: used to extract and aggregate classification labels from the input block embedding sequences of each frame through the Transformer coding layer; concatenate the aggregated classification labels with the corresponding block embedding sequences, and generate image features for each frame using the pre-trained CLIP image encoder; perform average pooling on the image features of each frame to obtain video features;

[0136] Text feature generation unit: Generates action category text features using an improved CLIP text encoder; the improved CLIP text encoder includes defining a fixed text template and inserting learnable cue words;

[0137] The classification decision unit classifies video actions based on the similarity between video features and text features.

[0138] In addition, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein when the processor executes the program, it implements the zero-shot video action recognition method based on multimodal cue learning as described above.

[0139] Alternatively, a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the zero-shot video action recognition method based on multimodal cue learning as described above.

[0140] Furthermore, in order to evaluate the performance of the proposed zero-shot video action recognition model GVU-MPT based on multimodal cue learning, this application selected 10 classic methods from existing zero-shot action recognition technologies, including Action CLIP, A5, AIM, ST-Adapter, X-CLIP, and Vita-CLIP, for comparison.

[0141] Implementation details include:

[0142] The algorithm was implemented using a deep learning framework based on Python 3.8.18 and PyTorch 1.12.1, with an NVIDIA GeForce RTX 4090 GPU (24GB VRAM) as the computing hardware configuration. For the experimental setup, the AdamW optimizer was used uniformly, with a base learning rate of 0.0004 and a weight decay coefficient of 0.001. To optimize the training process, a cosine annealing learning rate scheduler (CosineLRScheduler) was introduced, and a learning rate preheating strategy was implemented in the early stages of training.

[0143] The initial learning rate is set to 1×10. -6 (warmup_lr_init = 1.e-6), the warm-up period is 5 epochs, to stabilize gradient updates during the initial training phase of the model. The temperature coefficient τ in all experiments follows the default setting of CLIP, fixed at 0.01. In the balanced contextual cue fine-tuning method proposed in this invention, the length of the learnable cue words is uniformly set to 10.

[0144] Regarding the specific experimental setup:

[0145] 1) In the zero-shot generalization video action recognition experiment, four RTX 4090 GPUs were used for parallel training, the batch size was set to 2, and LoRA adaptation technology (rank r=4) was applied;

[0146] 2) In the cross-dataset generalization experiment, 8 RTX 4090 GPUs were used for training, the batch size was increased to 4, and the rank of the LoRA adaptation technique was adjusted to 8.

[0147] I. Zero-shot generalization experiment:

[0148] To analyze the zero-shot generalization capability of the GVU-MPT proposed in this invention and to make a fair comparison with various existing methods, this application follows the segmentation and evaluation protocol proposed in VIFI and performs generalization evaluation on four datasets: Kinetics-400, HMBD-51, UCF101, and SSv2.

[0149] For each dataset, all action categories are divided into two equal parts: the action category with more samples is designated as the visible class, and the action category with fewer samples is designated as the unseen class. Then, from the original training set, 16 video samples are randomly selected from each visible action class to form the training set for the experiment. All visible class videos from the original validation set are used as the validation set, and all unseen class videos from the original validation set are used as the test set. Furthermore, each video is limited to 32 frames, and a single-view inference paradigm is followed.

[0150] The model is first trained on the visible class and then evaluated on the unseen class. Experimental results report the top-1 accuracy for both the visible and unseen classes, and calculate their harmonic mean to test the fit, generalization, and overall performance of each model. The formula for calculating the top-1 accuracy is:

[0151]

[0152] Tables 1 and 2 show the zero-shot video action recognition performance of the proposed GVU-MPT and other models. "Seen" represents the model's classification accuracy on visible actions, which represents the model's fitting performance; "Unseen" represents the model's classification accuracy on unseen actions, which represents the model's generalization performance; HM (Harmonic Mean) is the harmonic mean of Seen and Unseen, which represents the overall performance of the model.

[0153] Table 1

[0154]

[0155] Table 2

[0156]

[0157] For each dataset, the best performance of Seen and Unseen is shown in bold in Tables 1 and 2. The performance evaluation results of the zero-shot generalization experiments of this invention on four commonly used video benchmark datasets are shown in Tables 1 and 2.

[0158] First, compared with the standard frozen pre-trained CLIP model without any modifications, the present invention shows a significant improvement in the accuracy of visible and unseen frequency action recognition, which indicates that the proposed GVU-MPT successfully extends the generalization ability of CLIP to the field of video recognition.

[0159] Secondly, methods like A5, ActionCLIP, and XCLIP learn temporal information and inter-frame relationships by introducing additional post-processing structures. These methods inevitably overfit to visible data during downstream training, causing the model to partially forget knowledge related to unseen actions. Specifically, this manifests in performance as lower accuracy in recognizing unseen video actions compared to a standard, frozen, pre-trained CLIP model without any modifications. The key reason for this is the relatively delayed timing of inter-frame information interaction; the image features extracted by the CLIP image encoder are already high-level conceptual features. Maintaining good generalization while aggregating information from each frame during information interaction between these highly abstract conceptual features is challenging.

[0160] Compared to post-processing-based methods like A5, ActionCLIP, and XCLIP, GVU-MPT exhibits a significant performance advantage. Compared to intermediate-structure-based methods like AIM and ST-Adapter, GVU-MPT's generalizable spatiotemporal modeling method, based on cross-frame cue fine-tuning, minimizes the impact on the gradient flow of the frozen CLI for all extensions to the frozen CLI. Furthermore, it incorporates a reused shared parameter design to mitigate overfitting in the introduced LoRA, resulting in a more ingenious and gentler overall approach. Therefore, GVU-MPT improves the accuracy of visible class recognition while maintaining the model's generalization ability to unseen classes, surpassing these methods in fitting performance, generalization performance, and overall performance. Finally, compared to the VIFI method, which performs full fine-tuning of the pre-trained model weights, GVU-MPT achieves a 4.3% and 9.2% improvement in harmonic mean accuracy, respectively, in the standard HMDB-51 and UCF-101 benchmarks, demonstrating a significant advantage. Meanwhile, this method significantly reduces training costs by freezing all CLIP pre-trained weights and only fine-tuning a small number of newly added parameters.

[0161] II. Cross-dataset generalization experiment

[0162] This invention strictly adheres to the cross-dataset generalization experiment partitioning criteria and evaluation protocol used in the VIFI method. Under the cross-dataset generalization setting, the model is first trained on the source dataset and then directly transferred to the target dataset for testing and evaluation. It is important to note that, to ensure the rigor of the evaluation, the action categories contained in the source and target datasets are completely non-overlapping. Specifically, this experiment first trains on the Kinetics-400 dataset, and then performs zero-shot transfer performance evaluation on three target datasets: HMDB-51, UCF-101, and Kinetics-600. For the HMDB-51 and UCF-101 datasets, the method is evaluated on their respective three validation set partitions, and the top-1 average classification accuracy and standard deviation are reported. For the Kinetics-600 dataset, the method is evaluated according to the literature.

[104] The evaluation protocol selects only 220 action categories that do not overlap with the Kinetics-400 action categories as the test category range. Three rounds of sampling are performed, each time drawing video samples from 160 of the 220 action categories, calculating the top-1 classification accuracy, and taking the average as the evaluation result. During the inference phase, all experiments use a single-view input strategy, with 32 frames of video samples as model input. The cross-dataset generalization experiment results are shown in Table 3.

[0163] Table 3

[0164] method HMDB-51 UCF-101 K-600 ER-ZSAR 35.3±4.6 51.8±2.9 42.1±1.4 Frozen CLIP 40.8±0.3 63.2±0.2 59.8±0.3 CLIPimage-FT 49.0±0.3 72.9±0.8 62.4±1.0 CLIPtext-FT 48.5±0.1 69.8±1.1 68.5±1.2 VIFI-CLIP 51.3±0.6 76.8±0.7 71.2±1.0 ActionCLIP 40.8±5.4 58.3±3.4 66.7±1.1 XCLIP 44.6±5.2 72.0±2.3 65.2±0.4 A5 44.3±2.2 69.3±4.2 55.8±0.7 AIM 50.3±0.8 79.4±1.0 66.7±0.5 ST-Adapter 51.1±0.6 77.6±0.7 60.2±1.8 Vita-CLIP 48.6±0.6 75.0±0.6 67.4±0.5 GVU-MPT (This invention) 54.1±0.8 85.5±0.7 73.1±0.7

[0165] Experimental results show that the native CLIP model exhibits significantly better initial generalization performance than similar multimodal models such as ER-ZSAR in direct zero-shot evaluation. This result verifies that the visual-language pre-trained model CLIP contains rich prior knowledge of human actions. Further research reveals that ViFi-CLIP, employing a fully fine-tuned strategy, achieves deep alignment between visual and textual representations in video tasks, resulting in significantly improved generalization ability compared to the parameter-frozen CLIP. Notably, although ViFi-CLIP is prone to overfitting and knowledge forgetting in the aforementioned zero-shot generalization experiments due to its large number of training parameters, the large-scale training data in cross-dataset generalization experiments based on the Kinetics-400 dataset (containing 240,000 training videos) alleviates the overfitting problem to some extent. This allows its performance to surpass posterior-structure-based methods such as A5 and ActionCLIP by 4.5%-18.5%, and to remain comparable to intermediate-structure-based methods such as AIM and ST-Adapter. This study proposes GVU-MPT, which gently extends CLIP through a generalizable spatiotemporal modeling method based on cross-frame cue fine-tuning in the visual branch and a balanced context cue fine-tuning method in the text branch. Experimental data show that GVU-MPT achieves a top-1 accuracy of 85.5% on the UCF-101 dataset, an improvement of 8.7% over ViFi-CLIP, effectively preserving CLIP's original generalization ability and successfully achieving a balance between adaptation and knowledge preservation in video action recognition.

[0166] III. Ablation Experiment

[0167] To verify the impact of these three mechanisms on the performance of GVU-MPT, this invention conducted systematic ablation experiments under the setting of zero-sample generalization experiments to verify the effectiveness and generalization ability of the proposed method. This application constructs three variants for ablation comparison by progressively adding LoRA-based multimodal tuning (MTLoRA), cross-frame cue-based spatiotemporal modeling (CFPT), and balanced context prompt tuning (BCPT) to a frozen pre-trained CLIP without any modifications: (1) CLIP+MTLoRA: adding a LoRA-based multimodal tuning method to a frozen pre-trained CLIP; (2) CLIP+MTLoRA+CFPT: adding a cross-frame cue-based spatiotemporal modeling mechanism to (1); (3) CLIP+MTLoRA+BCPT: adding a balanced context prompt tuning mechanism to (1); and (4) CLIP+MTLoRA+CFPT+BCPT: the complete method GVU-MPT proposed in this paper.

[0168] The ablation experimental results of the GVU-MPT model are shown in Table 4.

[0169] Table 4

[0170]

[0171] Compared to directly using the frozen pre-trained CLIP model, the introduction of a LoRA-based multimodal fine-tuning method to directionally optimize the multi-head attention module effectively activates the rich prior knowledge of human actions inherent in the CLIP visual language pre-trained model, resulting in a significant performance improvement. Furthermore, when introducing a generalizable spatiotemporal modeling mechanism based on cross-frame cue fine-tuning or a balanced context cue fine-tuning mechanism, these methods exhibit good synergy with LoRA multimodal fine-tuning. Performance remains relatively stable on visible categories, but performance on unseen categories shows significant improvement, fully demonstrating that the proposed method has stronger generalization ability for zero-shot video action recognition. Finally, the complete GVU-MPT model integrates the above three innovative mechanisms, achieving optimal performance in zero-shot video action recognition tasks: a recognition accuracy of 81.70% on unseen categories, while also achieving a highest harmonic mean accuracy of 87.59%. Ablation experiments not only verify the effectiveness of each module design but also fully demonstrate the superiority of the GVU-MPT framework in zero-shot video action recognition tasks.

[0172] IV. Visualization Experiments and Analysis

[0173] To visually verify the improvement effect of GVU-MPT on the model's generalization ability, this invention randomly selected 10 unseen action categories on the UCF101 dataset for video feature space t-SNE visualization analysis. The visualization results are as follows: Figure 6 As shown, video features belonging to the same action category are represented by a uniform color, where (a) is the feature distribution map using the CLIP baseline method, and (b) is the feature distribution map using GVU-MPT. Comparing the feature distributions of the CLIP baseline method and GVU-MPT reveals that GVU-MPT exhibits a higher degree of clustering among similar video samples; the feature distributions among dissimilar video samples show a clearer discriminative structure. This result verifies that GVU-MPT effectively enhances the model's generalization ability through its multimodal cue learning mechanism.

[0174] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0175] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A zero-shot video action recognition method based on multimodal cue learning, characterized in that, Video feature extraction: The classification labels in the input block embedding sequence of each frame are extracted and aggregated through the Transformer coding layer; the aggregated classification labels are concatenated with the corresponding block embedding sequence, and the pre-trained CLIP image encoder is used to generate the image features of each frame; Average pooling is performed on the image features of each frame to obtain video features; Text feature generation: Generating action category text features using an improved CLIP text encoder; The improved CLIP text encoder includes defining fixed text templates and inserting learnable cue words; Video action classification is achieved based on the similarity between the video features and the text features.

2. The zero-sample video action recognition method according to claim 1, characterized in that, The aggregation process includes layer normalization of classification labels in the block embedding sequence using LayerNorm1 of the CLIP image encoder, and aggregation of cross-frame information using multi-head attention.

3. The zero-shot video action recognition method according to claim 1, characterized in that, The pre-trained CLIP image encoder is used to generate image features for each frame, including: After computing spatial self-attention, the classification label appended to the end of the embedding sequence is removed through slicing operations to make the dimension of the embedding sequence consistent with the original input.

4. The zero-sample video action recognition method according to claim 1 or 2, characterized in that, The multi-head self-attention modules in the CLIP image encoder and CLIP text encoder are fine-tuned based on LoRA, including: mapping the LoRA to the multi-head self-attention module's mapping matrix W. q W k W v W o In parallel, LoRA is decomposed using low-rank matrices A and B.

5. The zero-shot video action recognition method according to claim 1, characterized in that, The fixed text template is "aphoto of apersondoing[class]".

6. The zero-sample video action recognition method according to claim 1 or 5, characterized in that, Add N learnable prompts X at the beginning and end of the fixed text template to form the structure "XXX[fixed template]XXX".

7. The zero-sample video action recognition method according to claim 1, characterized in that, An improved CLIP text encoder is used to generate action category text features, including replacing the word embeddings of the cue word tag X with learnable vectors.

8. A zero-shot video action recognition system based on multimodal cue learning, characterized in that, The zero-shot video action recognition method based on multimodal cue learning as described in any one of claims 1-7 includes: Video feature extraction unit: used to extract and aggregate classification labels from the input block embedding sequences of each frame through the Transformer coding layer; concatenate the aggregated classification labels with the corresponding block embedding sequences, and generate image features for each frame using the pre-trained CLIP image encoder; perform average pooling on the image features of each frame to obtain video features; Text feature generation unit: Generates action category text features using an improved CLIP text encoder; the improved CLIP text encoder includes defining a fixed text template and inserting learnable cue words; The classification decision unit classifies video actions based on the similarity between video features and text features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the zero-shot video action recognition method based on multimodal cue learning as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the zero-shot video action recognition method based on multimodal cue learning as described in any one of claims 1-7.

Images