Zero sample video classification method based on test time visual proxy tuning

Abstract

According to the zero sample video classification method based on test time visual agent tuning provided by the invention, the visual agent is constructed by using the support set, and the visual prompt and the text prompt are finely tuned at the same time, so that the zero sample action recognition of the video action is realized, and the semantic gap problem between the visual mode and the text mode in the video classification task is avoided. And the visual agent construction module and the bimodal prompt collaborative tuning module form a zero sample learning framework TPC. A pre-trained video encoder is utilized to extract support set video features in a visual agent construction module to construct a visual agent, and learnable visual cues are added to sampled support set videos to enable the visual agent to be adjustable. In the bimodal prompt collaborative tuning module, KL divergence between probability distributions is predicted by minimizing a visual agent and a text agent, learnable visual prompts and text prompts are finely tuned at the same time, the visual agent is optimized by using information of the text mode and the visual mode, and the zero sample classification performance of the visual agent is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a zero-shot video classification method based on test-time visual agent tuning. Background Technology

[0002] With the development of deep learning, video classification research has achieved powerful supervised performance and has been successfully applied in many fields, including intelligent robots, security monitoring, and autonomous driving. However, existing methods rely on training on large-scale video classification datasets, resulting in a limited number of categories that can be recognized, making it difficult to meet the growing practical applications and demands. In recent years, the powerful zero-shot generalization ability of large-scale pre-trained visual language models (VLMs) has driven the rapid development of zero-shot video classification, and visual language models have become a hope for breaking through the perception limits of existing technologies. These visual language model-based methods utilize pre-trained visual language models to project test videos and category names into a joint feature space, and then assign labels to test videos based on the feature similarity between them, thereby achieving video classification.

[0003] Despite the massive scale of existing video classification datasets used for training, a distribution skew still exists between the training and testing sets. Therefore, it's necessary to further tune the parameters or cues of the pre-trained model for specific testing environments. Existing methods can be broadly categorized into two types: training-based methods and training-free methods. Training-based methods typically utilize large amounts of video data to fine-tune some parameters of the pre-trained model or add additional learnable cues, which requires significant training costs. Training-free methods, on the other hand, adjust the model's parameters or learnable cues in a self-supervised manner for each test video during testing. Existing test-time tuning methods often freeze the visual space, adjusting only the text proxy located in the text space. However, recent research has found that even after optimization, the gap between the text and visual spaces remains significant.

[0004] Therefore, how to construct a suitable visual agent in the visual space and further optimize the visual agent to effectively identify unknown categories is a key challenge in zero-shot video classification. Summary of the Invention

[0005] To address the low accuracy of existing technologies in identifying unknown categories, this invention provides a zero-shot video classification method based on time-of-test (TOT) visual agent tuning. Zero-shot video classification is achieved through the TOT learning framework, TPC. This framework employs a learning paradigm combining a visual agent construction module and a bimodal cue collaborative tuning module, thus resolving the issue of semantic distribution discrepancies between text and visual modalities in zero-shot video classification.

[0006] Technical Solution: A zero-shot video classification method based on a visual agent construction module and a bimodal cue collaborative tuning module. This method constructs and optimizes a visual agent located in the visual space through the visual agent construction module and the bimodal cue collaborative tuning module, enabling video classification in a zero-shot environment and avoiding the semantic gap problem between different modalities in zero-shot tasks. The visual agent construction module and the bimodal cue collaborative tuning module constitute the zero-shot learning framework TPC. The implementation of the method includes the following steps:

[0007] Step 1: For a test video and a given set of action categories, combine the learnable text prompts with each action category to form a category description text, and use a pre-trained visual language model to extract the visual features of the test video and the text features of the category description text.

[0008] Step 2: For a given set of action categories, generate multiple different action descriptions for each action category using a Large Language Model (LLM); then use the generated action descriptions to generate a batch of videos for each action category using a Text-to-Video (T2V) model; next, use a pre-trained visual language model to calculate the score of the real category to which the generated videos belong, and select the K videos with the highest scores for each action category to construct a support set.

[0009] Step 3: Pass the support set into the visual agent building module to obtain the visual agent for prediction.

[0010] Step 4: Utilize the bimodal cue co-tuning module to simultaneously adjust the learnable text and visual cues, thereby optimizing the visual agent for different test videos.

[0011] Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification.

[0012] Furthermore, step 1 includes the following specific steps:

[0013] For input video V test The T sampled images are used to extract corresponding visual features of dimension d using a pre-trained visual language model video encoder.

[0014]

[0015] Where E v It is a video encoder based on a pre-trained visual language model;

[0016] For an action category set Y = {y0, y1, ..., y} with C categories, c ,} will provide learnable text prompts p tEach action category is combined with a category description text, and the corresponding text features of dimension d are extracted using a pre-trained visual language model's text encoder as text proxies.

[0017]

[0018] Where E t It is a text encoder for a pre-trained visual language model.

[0019] Furthermore, the support set construction process is as follows:

[0020] Step 201: For any action category y i Each action is transformed into m different action descriptions using a large language model, namely:

[0021]

[0022] Where LLM(·) is the Large Language Model, This represents the j-th description of the i-th action category;

[0023] Step 202: For a set of action categories Each action category is processed in step 201 to generate an action description set with C×m action descriptions.

[0024] Step 203: For action category y in the action category set i Using a text-to-video generation model based on action description set D, y i The corresponding m action descriptions are used to generate k video samples;

[0025] Step 204: For category y i For the corresponding k generated video samples, a pre-trained visual language model is used to calculate the category of each generated video sample in y. i Based on the probability scores, select the K video samples with the highest scores;

[0026] Step 205: By generating and selecting video samples for each action category using steps 203 and 204, a support set S with C×K supporting videos is finally constructed.

[0027] Furthermore, the execution process of the visual agent construction module is as follows:

[0028] Step 301: For each supporting video in the support set S, sample T. S Zhang images, which will provide learnable visual cues p v Add it to the sampled image to obtain the supported set of video images;

[0029] Step 302: Extract visual features from the sampled support set video images using a pre-trained video encoder and stitch them together to form a visual proxy.

[0030]

[0031] Where F i v represents the visual feature that supports the i-th video. i This indicates the i-th supported video in the support set. Concat(·) represents the vector concatenation operation. C and K represent the number of categories and the number of support set samples corresponding to each category, respectively. d is the feature dimension.

[0032] Furthermore, the execution process of the bimodal prompting collaborative tuning module is as follows:

[0033] Step 401: For each test video V test Aug is augmented M times using the AugMix data augmentation method, resulting in M ​​augmented views Aug. i (V test ), i∈[1,M];

[0034] Step 402: Extract visual features from the M enhanced views obtained in step 401 using a pre-trained video encoder.

[0035]

[0036] in, This represents the visual features of the i-th augmented view of the test video;

[0037] Step 403: Compare the visual features of the enhanced test video view obtained in step 402 with the visual proxy Z obtained in step 302. v Calculate the probability distribution p predicted by the visual agent for each augmented view of the test video. vp (Y|Aug i (V test )):

[0038]

[0039] Where Softmax(·) represents the Softmax function operation, τ v β is the temperature parameter used by the visual agent during Softmax calculations, and β is a parameter that adjusts sharpness. Support set tags for one-hot encoding;

[0040] Step 404: Compare the visual features of the enhanced test video view obtained in step 402 with the text proxy Z obtained in step 1. t Calculate the probability distribution p predicted by the text agent for each augmented view of the test video. tp (Y|Aug i (V test )):

[0041]

[0042] Where τ t This is the temperature parameter used by the text agent when performing Softmax calculations;

[0043] Step 405: For each augmented view of the calculated test video, predict the probability distribution p using the visual agent. vp (Y|Aug i (V test )) and the probability distribution p predicted using a text proxy tp (Y|Aug i (V test This method utilizes the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution to dynamically adjust the learnable visual cues p for the test video during testing. v and text prompt p t :

[0044]

[0045] Among them, D KL (·) represents the KL divergence calculation between two probabilistic texts;

[0046] Step 406: Obtain the fine-tuned learnable visual cue by minimizing the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution. Obtain the final visual agent used for prediction

[0047]

[0048] in, Indicates in The visual feature of the i-th supporting video is extracted with the assistance of [unclear].

[0049] Finally, for the visual agent obtained in step 406 for prediction Calculate the predicted score of the test video This allows for zero-shot action recognition, i.e.:

[0050]

[0051] This application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the zero-shot video classification method based on test-time visual agent tuning provided in this application.

[0052] This application also provides a computer-readable storage medium storing a computer program that executes the zero-shot video classification method based on test-time visual agent tuning provided in this application.

[0053] Beneficial Effects: The zero-shot video classification method based on test-time visual agent tuning proposed in this invention achieves state-of-the-art results on four datasets: HMDB-51, UCF-101, Kinetics-600, and ActivityNet. This method constructs and optimizes a visual agent in the visual space through a visual agent construction module and a bimodal cue collaborative tuning module, enabling video classification in a zero-shot environment and avoiding the semantic gap between text and visual modalities in zero-shot tasks. The visual agent construction module utilizes the support set to construct the visual agent and adds learnable visual cues to sampled support set video frames to make the visual agent tunable. The bimodal cue collaborative tuning module, under the supervision of the consistency of prediction distributions of text and visual agents, adjusts the learnable text and visual cues in multiple steps, thereby extracting key information from both text and visual modalities to aid inference. Ultimately, state-of-the-art performance is achieved in zero-shot video classification tasks. Attached Figure Description

[0054] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0057] To address the semantic gap between the visual and text modalities in zero-shot video classification tasks, this invention employs a zero-shot learning framework, TPC, based on test-time visual proxy tuning. A text-to-video generation model uses a large language model to generate a batch of videos for each category, and a prediction score selection mechanism selects the K videos with the highest predicted scores for the true category from the generated videos for each category to form a support set. Learnable visual cues are added to the sampled support set video frames, and a pre-trained video encoder extracts features from the support set videos to construct a visual proxy. Visual proxy tuning is achieved by minimizing the KL divergence between the text and visual proxy prediction distributions. This approach avoids the gap between the text and visual modalities and improves the model's ability to recognize unknown video categories.

[0058] like Figure 1 This paper presents a zero-shot video classification method based on time-of-test (TOT) visual proxy tuning. Through a visual proxy construction module and a bimodal cue collaborative tuning module, it achieves zero-shot video classification, avoiding the semantic gap between text and visual modalities in video classification tasks. The visual proxy construction module and the bimodal cue collaborative tuning module constitute the zero-shot learning framework (TPC). In the visual proxy construction module, a pre-trained video encoder is used to extract support set video features to construct the visual proxy, and learnable visual cues are added to the sampled support set videos to make the visual proxy tunable. In the bimodal cue collaborative tuning module, the learnable visual and text cues are fine-tuned simultaneously by minimizing the KL divergence between the predicted probability distributions of the visual and text proxies. Information from both the text and visual modalities is used to optimize the visual proxy, improving its zero-shot classification performance and avoiding the problems caused by the gap between text and visual modalities. Specifically, the method includes the following steps:

[0059] The execution process of the dual-modal prompting collaborative tuning module is as follows:

[0060] Step 1: For the input video V test The T sampled images are used to extract corresponding visual features of dimension d from the video encoder of a pre-trained visual language model.

[0061]

[0062] Where E v It is a video encoder based on a pre-trained visual language model.

[0063] For an action category set Y = {y0, y1, ..., y} with C categories, c ,} will provide learnable text prompts p tEach action category is combined with a category description text, and the corresponding text features of dimension d are extracted using a pre-trained visual language model's text encoder as text proxies.

[0064]

[0065] Where E t It is a text encoder for a pre-trained visual language model.

[0066] Step 2: For any action category y i Each action is transformed into m different action descriptions using a large language model, namely:

[0067]

[0068] Where LLM(·) is the Large Language Model, This represents the j-th description of the i-th action category.

[0069] Step 3: For a set of action categories Each action category is processed in step 201 to generate an action description set with C×m action descriptions.

[0070] Step 4: For action category y in the action category set i Using a text-to-video generation model based on action description set D, y i The corresponding m action descriptions are used to generate k video samples.

[0071] Step 5: For category y i For the corresponding k generated video samples, a pre-trained visual language model is used to calculate the category of each generated video sample in y. i Based on the probability scores, select the K video samples with the highest scores.

[0072] Step 6: By generating and selecting video samples for each action category using steps 4 and 5, a support set S with C×K supporting videos is finally constructed.

[0073] Step 7: For each supporting video in the support set S constructed in Step 6, sample T. S Zhang images, which will provide learnable visual cues p v Add to the sampled image.

[0074] Step 8: For the support set video images obtained after sampling in Step 7, extract their visual features using a pre-trained video encoder and stitch them together to form a visual proxy.

[0075]

[0076] Where F i v represents the visual feature that supports the video. i This represents the i-th supported video in the support set. Concat(·) represents the vector concatenation operation. C and K represent the number of categories and the number of support set samples corresponding to each category, respectively. d is the feature dimension.

[0077] Step 9: For each test video V test The AugMix data augmentation method is used to augment the data M times, resulting in K augmented views Avg. i (V test ), i∈[1,M.

[0078] Step 10: Extract visual features from the M enhanced views obtained in Step 9 using a pre-trained video encoder:

[0079]

[0080] in, This represents the visual features of the i-th augmented view of the test video.

[0081] Step 11: Compare the visual features of the enhanced test video view obtained in Step 10 with the visual proxy Z obtained in Step 8. v Calculate the probability distribution p predicted by the visual agent for each augmented view of the test video. vp (Y|Aug i (V test )):

[0082]

[0083] Where Softmax(·) represents the Softmax function operation, τ v This is the temperature parameter used by the visual agent during Softmax calculations, and β is a parameter that adjusts the "sharpness". Support set tags for one-hot encoding.

[0084] Step 12: Compare the visual features of the enhanced test video view obtained in Step 10 with the text proxy Z obtained in Step 1. t Calculate the probability distribution p predicted by the text agent for each augmented view of the test video. tp (Y|Aug i (V test )):

[0085]

[0086] Where τ tThis is the temperature parameter used by the text agent when performing Softmax calculations.

[0087] Step 13: For each augmented view of the calculated test video, predict the probability distribution p using the visual agent. vp (Y|Aug i (V test )) and the probability distribution p predicted using a text proxy tp (Y|Aug i (V test This method utilizes the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution to dynamically adjust the learnable visual cues p for the test video during testing. v and text prompt p t :

[0088]

[0089] Among them, D KL (·) represents the KL divergence calculation between two probabilistic texts.

[0090] Step 14: Obtain the fine-tuned, learnable visual cues by minimizing the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution. Obtain the final visual agent used for prediction

[0091]

[0092] in, Indicates in The visual feature of the i-th supporting video is extracted with the assistance of [unclear].

[0093] Step 15: For the visual agent obtained in step 14 used for prediction Calculate the predicted score of the test video This allows for zero-shot action recognition, i.e.:

[0094]

[0095] Table 1 compares the results of our method with other zero-shot video classification methods on four video classification datasets: HMDB-51, UCF-101, Kinetics-600, and Activity Net, with TOP-1 accuracy as the evaluation metric.

[0096] Obviously, those skilled in the art should understand that the steps of the zero-shot video classification method based on test-time visual agent tuning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0097] Table 1 Comparison of results across the four datasets

[0098]

[0099] The embodiments described above do not constitute a limitation on the scope of protection of this application.

Claims

1. A zero-shot video classification method based on test-time visual proxy tuning, characterized in that The method comprises: Step 1: For a test video and a given set of action categories, the learnable text prompts and each action category are combined into a category description text, and the pre-trained visual language model is used to extract the visual features of the test video and the text features of the category description text respectively; Step 2: For a given set of action categories, a large language model (LLM) is used to generate multiple different action descriptions for each action category. Then, a batch of videos are generated for each action category using the generated action descriptions through the text-to-video generation model (T2V). Next, the pre-trained visual language model is used to calculate the scores of the true categories to which the generated videos belong, and the K videos with the highest scores are selected for each action category to construct the support set. Step 3: Pass the support set into the visual agent building module to obtain the visual agent for prediction; Step 4: Use the bimodal cue co-tuning module to simultaneously adjust the learnable text and visual cues to optimize the visual agent for different test videos; Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification.

2. The method according to claim 1, characterized in that Step 1 includes: For the input video V test The sampled T images use the video encoder of the pre-trained visual language model to extract the visual features corresponding to the feature dimension d Where E v It is a video encoder with pre-trained visual language model; For an action category set Y with C categories, c ,}, the learnable text prompt p t The category description text is composed with each action category, and the text encoder of the pre-trained visual language model is used to extract the text features with the corresponding feature dimension d as the text proxy Where E t It is a text encoder of a pre-trained visual language model.

3. The method according to claim 2, characterized in that The support set construction process is as follows: Step 201: For any action category y i , each action is converted into m different action descriptions through a large language model, namely: Where LLM(·) is the large language model, represents the jth description of the i-th action category; Step 202: For an action category set Each action category is processed in step 201 to perform action description, and finally an action description set having C×m action descriptions is generated. Step 203: For the action category y in the action category set i , use the text-to-video generation model to generate k video samples according to the m action descriptions corresponding to yi in the action description set D; Step 204: for the k generated video samples corresponding to category yi, use the pre-trained visual language model to calculate the probability score of each generated video sample belonging to yi, and select the K video samples with the highest scores; Step 205: By generating and selecting video samples for each action category using steps 203 and 204, a support set S with C×K support videos is finally constructed.

4. The method according to claim 3, characterized in that The execution process of the visual agent building module is as follows: Step 301: For each support video in the support set S, sample T S images, and the learnable visual cues p v Add to the sampled image to get the support set video image; Step 302: Extract visual features from the sampled support set video images using a pre-trained video encoder and concatenate them to form a visual proxy. where F i represents the visual features of the i-th supporting video, v i represents the i-th support video in the support set, Concat(·) represents the concatenation operation of vectors, C and K represent the number of categories and the number of support set samples corresponding to each category, respectively, and d is the feature dimension.

5. The method according to claim 4, characterized in that The execution process of the dual-modal prompt collaborative tuning module is as follows: Step 401: For each test video V test , using the AugMix data enhancement method to enhance M times, and obtain M enhanced views Aug i (V test ), i∈[1,M]; Step 402: For the M enhanced views obtained in step 401, use a pre-trained video encoder to extract visual features: in, represents the visual features of the i-th enhanced view of the test video; Step 403: Compare the visual features of the test video enhanced view obtained in step 402 with the visual proxy Z obtained in step 302. v , calculate the probability distribution p predicted by the visual agent for each enhanced view of the test video vp (Y|Aug i (V test )): Where Softmax(·) represents the Softmax function operation, τ v is the temperature parameter used by the visual agent for Softmax calculation, and β is the parameter for adjusting sharpness. is the support set label for one-hot encoding; Step 404: Compare the visual features of the test video enhanced view obtained in step 402 with the text proxy Z obtained in step 1 t , calculate the probability distribution p predicted by the text agent for each enhanced view of the test video tp (Y|Aug i (V test )): where τ t It is the temperature parameter used by the text agent for Softmax calculation; Step 405: For each enhanced view of the calculated test video, use the probability distribution p predicted by the visual agent vp (Y|Aug i (V test )) and the probability distribution p predicted using the text proxy tp (Y|Aug i (V test ), the learnable visual cues p are dynamically adjusted for the test video during testing by minimizing the KL divergence between the visual proxy prediction probability distribution and the text proxy prediction probability distribution. v and text prompt p t : Among them, D KL (·) represents the KL divergence calculation between two probability texts; Step 406: Obtain a fine-tuned learnable visual cue by minimizing the KL divergence between the visual agent prediction probability distribution and the text agent prediction probability distribution Obtain the final visual proxy for prediction in, Indicated in The visual features of the i-th supporting video are extracted with the assistance of .

6. The method according to claim 5, characterized in that Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification, including: Finally, for the visual agent used for prediction obtained in step 406 Calculate the prediction score for the test video So as to perform zero-sample action recognition, that is:

Images