Zero-shot video classification method based on test time visual proxy tuning

Abstract

The application provides a zero-shot video classification method based on test time visual proxy tuning, which realizes zero-shot action recognition of video actions by constructing a visual proxy using a support set and simultaneously fine-tuning visual and text prompts, and avoids the semantic gap problem between the two modalities of video and text in the video classification task. The visual proxy construction module and the dual-modal prompt collaborative tuning module constitute a zero-shot learning framework TPC. In the visual proxy construction module, a pre-trained video encoder is used to extract support set video features to construct a visual proxy, and a learnable visual prompt is added to the sampled support set video to make the visual proxy adjustable. In the dual-modal prompt collaborative tuning module, the learnable visual and text prompts are fine-tuned by minimizing the KL divergence between the prediction probability distribution of the visual proxy and the text proxy, the information of the text and visual modalities is used to optimize the visual proxy, and the zero-shot classification performance of the visual proxy is improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a zero-shot video classification method based on test-time visual agent tuning. Background Technology

[0002] With the development of deep learning, video classification research has achieved powerful supervised performance and has been successfully applied in many fields, including intelligent robots, security monitoring, and autonomous driving. However, existing methods rely on training on large-scale video classification datasets, resulting in a limited number of categories that can be recognized, making it difficult to meet the growing practical applications and demands. In recent years, the powerful zero-shot generalization ability of large-scale pre-trained visual language models (VLMs) has driven the rapid development of zero-shot video classification, and visual language models have become a hope for breaking through the perception limits of existing technologies. These visual language model-based methods utilize pre-trained visual language models to project test videos and category names into a joint feature space, and then assign labels to test videos based on the feature similarity between them, thereby achieving video classification.

[0003] Despite the massive scale of existing video classification datasets used for training, a distribution skew still exists between the training and testing sets. Therefore, it's necessary to further tune the parameters or cues of the pre-trained model for specific testing environments. Existing methods can be broadly categorized into two types: training-based methods and training-free methods. Training-based methods typically utilize large amounts of video data to fine-tune some parameters of the pre-trained model or add additional learnable cues, which requires significant training costs. Training-free methods, on the other hand, adjust the model's parameters or learnable cues in a self-supervised manner for each test video during testing. Existing test-time tuning methods often freeze the visual space, adjusting only the text proxy located in the text space. However, recent research has found that even after optimization, the gap between the text and visual spaces remains significant.

[0004] Therefore, how to construct a suitable visual agent in the visual space and further optimize the visual agent to effectively identify unknown categories is a key challenge in zero-shot video classification. Summary of the Invention

[0005] To address the low accuracy of existing technologies in identifying unknown categories, this invention provides a zero-shot video classification method based on time-of-test (TOT) visual agent tuning. Zero-shot video classification is achieved through the TOT learning framework, TPC. This framework employs a learning paradigm combining a visual agent construction module and a bimodal cue collaborative tuning module, thus resolving the issue of semantic distribution discrepancies between text and visual modalities in zero-shot video classification.

[0006] Technical Solution: A zero-shot video classification method based on a visual agent construction module and a bimodal cue collaborative tuning module. This method constructs and optimizes a visual agent located in the visual space through the visual agent construction module and the bimodal cue collaborative tuning module, enabling video classification in a zero-shot environment and avoiding the semantic gap problem between different modalities in zero-shot tasks. The visual agent construction module and the bimodal cue collaborative tuning module constitute the zero-shot learning framework TPC. The implementation of the method includes the following steps:

[0007] Step 1: For a test video and a given set of action categories, combine the learnable text prompts with each action category to form a category description text, and use a pre-trained visual language model to extract the visual features of the test video and the text features of the category description text.

[0008] Step 2: For a given set of action categories, generate multiple different action descriptions for each action category using a Large Language Model (LLM); then use the generated action descriptions to generate a batch of videos for each action category using a Text-to-Video (T2V) model; next, use a pre-trained visual language model to calculate the score of the real category to which the generated videos belong, and select the K videos with the highest scores for each action category to construct a support set.

[0009] Step 3: Pass the support set into the visual agent building module to obtain the visual agent for prediction.

[0010] Step 4: Utilize the bimodal cue co-tuning module to simultaneously adjust the learnable text and visual cues, thereby optimizing the visual agent for different test videos.

[0011] Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification.

[0012] Furthermore, step 1 includes the following specific steps:

[0013] For input video V test The T sampled images are used to extract corresponding visual features of dimension d using a pre-trained visual language model video encoder.

[0014]

[0015] Where E v It is a video encoder based on a pre-trained visual language model;

[0016] For an action category set Y = {y0, y1, ..., y} with C categories, c ,}, will provide learnable text prompts p tEach action category is combined with a category description text, and the corresponding text features of dimension d are extracted using a pre-trained visual language model's text encoder as text proxies.

[0017]

[0018] Where E t It is a text encoder for a pre-trained visual language model.

[0019] Furthermore, the support set construction process is as follows:

[0020] Step 201: For any action category y i Each action is transformed into m different action descriptions using a large language model, namely:

[0021]

[0022] Where LLM(·) is the Large Language Model, This represents the j-th description of the i-th action category;

[0023] Step 202: For a set of action categories Each action category is processed in step 201 to generate an action description set with C×m action descriptions.

[0024] Step 203: For action category y in the action category set i Using a text-to-video generation model based on action description set D, y i The corresponding m action descriptions are used to generate k video samples;

[0025] Step 204: For category y i For the corresponding k generated video samples, a pre-trained visual language model is used to calculate the category of each generated video sample in y. i Based on the probability scores, select the K video samples with the highest scores;

[0026] Step 205: By generating and selecting video samples for each action category using steps 203 and 204, a support set S with C×K supporting videos is finally constructed.

[0027] Furthermore, the execution process of the visual agent construction module is as follows:

[0028] Step 301: For each supporting video in the support set S, sample T. S Zhang images, which will provide learnable visual cues p v Add it to the sampled image to obtain the supported set of video images;

[0029] Step 302: Extract visual features from the sampled support set video images using a pre-trained video encoder and stitch them together to form a visual proxy.

[0030]

[0031] Where F i v represents the visual feature that supports the video. i This represents the i-th supported video in the support set. Concat(·) represents the vector concatenation operation. C and K represent the number of categories and the number of support set samples for each category, respectively. d is the feature dimension.

[0032] Furthermore, the execution process of the bimodal prompting collaborative tuning module is as follows:

[0033] Step 401: For each test video V test Aug is augmented M times using the AugMix data augmentation method, resulting in M ​​augmented views Aug. i (V test ), i∈[1,M];

[0034] Step 402: Extract visual features from the M enhanced views obtained in step 401 using a pre-trained video encoder.

[0035]

[0036] in, This represents the visual features of the i-th augmented view of the test video;

[0037] Step 403: Compare the visual features of the enhanced test video view obtained in step 402 with the visual proxy Z obtained in step 302. v Calculate the probability distribution p predicted by the visual agent for each augmented view of the test video. vp (Y|Aug i (V test )):

[0038]

[0039] Where Softmax(·) represents the Softmax function operation, τ v β is the temperature parameter used by the visual agent during Softmax calculations, and β is a parameter that adjusts sharpness. Support set tags for one-hot encoding;

[0040] Step 404: Compare the visual features of the enhanced test video view obtained in step 402 with the text proxy Z obtained in step 1. t Calculate the probability distribution p predicted by the text agent for each augmented view of the test video. tp (Y|Aug i (V test )):

[0041]

[0042] Where τ t This is the temperature parameter used by the text agent when performing Softmax calculations;

[0043] Step 405: For each augmented view of the calculated test video, predict the probability distribution p using the visual agent. vp (Y|Aug i (V test )) and the probability distribution p predicted using a text proxy tp (Y|Aug i (V test This method utilizes the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution to dynamically adjust the learnable visual cues p for the test video during testing. v and text prompt p t :

[0044]

[0045] Among them, D KL (·) represents the KL divergence calculation between two probabilistic texts;

[0046] Step 406: Obtain the fine-tuned learnable visual cue by minimizing the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution. Obtain the final visual agent used for prediction

[0047]

[0048] in, Indicates in The visual feature of the i-th supporting video is extracted with the assistance of [unclear].

[0049] Finally, for the visual agent obtained in step 406 for prediction Calculate the predicted score of the test video This allows for zero-shot action recognition, i.e.:

[0050]

[0051] This application also provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the zero-shot video classification method based on test-time visual agent tuning provided in this application.

[0052] This application also provides a computer-readable storage medium storing a computer program that executes the zero-shot video classification method based on test-time visual agent tuning provided in this application.

[0053] Beneficial Effects: The zero-shot video classification method based on test-time visual agent tuning proposed in this invention achieves state-of-the-art results on four datasets: HMDB-51, UCF-101, Kinetics-600, and ActivityNet. This method constructs and optimizes a visual agent in the visual space through a visual agent construction module and a bimodal cue collaborative tuning module, enabling video classification in a zero-shot environment and avoiding the semantic gap between text and visual modalities in zero-shot tasks. The visual agent construction module utilizes the support set to construct the visual agent and adds learnable visual cues to sampled support set video frames to make the visual agent tunable. The bimodal cue collaborative tuning module, under the supervision of the consistency of prediction distributions of text and visual agents, adjusts the learnable text and visual cues in multiple steps, thereby extracting key information from both text and visual modalities to aid inference. Ultimately, state-of-the-art performance is achieved in zero-shot video classification tasks. Attached Figure Description

[0054] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] The embodiments of this application will now be described in conjunction with the accompanying drawings.

[0057] To address the semantic gap between the visual and text modalities in zero-shot video classification tasks, this invention employs a zero-shot learning framework, TPC, based on test-time visual proxy tuning. A text-to-video generation model uses a large language model to generate a batch of videos for each category, and a prediction score selection mechanism selects the K videos with the highest predicted scores for the true category from the generated videos for each category to form a support set. Learnable visual cues are added to the sampled support set video frames, and a pre-trained video encoder extracts features from the support set videos to construct a visual proxy. Visual proxy tuning is achieved by minimizing the KL divergence between the text and visual proxy prediction distributions. This approach avoids the gap between the text and visual modalities and improves the model's ability to recognize unknown video categories.

[0058] like Figure 1 This paper presents a zero-shot video classification method based on time-of-test (TOT) visual proxy tuning. Through a visual proxy construction module and a bimodal cue collaborative tuning module, it achieves zero-shot video classification, avoiding the semantic gap between text and visual modalities in video classification tasks. The visual proxy construction module and the bimodal cue collaborative tuning module constitute the zero-shot learning framework (TPC). In the visual proxy construction module, a pre-trained video encoder is used to extract support set video features to construct the visual proxy, and learnable visual cues are added to the sampled support set videos to make the visual proxy tunable. In the bimodal cue collaborative tuning module, the learnable visual and text cues are fine-tuned simultaneously by minimizing the KL divergence between the predicted probability distributions of the visual and text proxies. Information from both the text and visual modalities is used to optimize the visual proxy, improving its zero-shot classification performance and avoiding the problems caused by the gap between text and visual modalities. Specifically, the method includes the following steps:

[0059] The execution process of the dual-modal prompting collaborative tuning module is as follows:

[0060] Step 1: For the input video V test The T sampled images are used to extract corresponding visual features of dimension d from the video encoder of a pre-trained visual language model.

[0061]

[0062] Where E v It is a video encoder based on a pre-trained visual language model.

[0063] For an action category set Y = {y0, y1, ..., y} with C categories, c ,}, will provide learnable text prompts p tEach action category is combined with a category description text, and the corresponding text features of dimension d are extracted using a pre-trained visual language model's text encoder as text proxies.

[0064]

[0065] Where E t It is a text encoder for a pre-trained visual language model.

[0066] Step 2: For any action category y i Each action is transformed into m different action descriptions using a large language model, namely:

[0067]

[0068] Where LLM(·) is the Large Language Model, This represents the j-th description of the i-th action category.

[0069] Step 3: For a set of action categories Each action category is processed in step 201 to generate an action description set with C×m action descriptions.

[0070] Step 4: For action category y in the action category set i Using a text-to-video generation model based on action description set D, y i The corresponding m action descriptions are used to generate k video samples.

[0071] Step 5: For category y i For the corresponding k generated video samples, a pre-trained visual language model is used to calculate the category of each generated video sample in y. i Based on the probability scores, select the K video samples with the highest scores.

[0072] Step 6: By generating and selecting video samples for each action category using steps 4 and 5, a support set S with C×K supporting videos is finally constructed.

[0073] Step 7: For each supporting video in the support set S constructed in Step 6, sample T. S Zhang images, which will provide learnable visual cues p v Add to the sampled image.

[0074] Step 8: For the support set video images obtained after sampling in Step 7, extract their visual features using a pre-trained video encoder and stitch them together to form a visual proxy.

[0075]

[0076] Where F i v represents the visual feature that supports the video. i This represents the i-th supported video in the support set. Concat(·) represents the vector concatenation operation. C and K represent the number of categories and the number of support set samples corresponding to each category, respectively. d is the feature dimension.

[0077] Step 9: For each test video V test The AugMix data augmentation method is used to augment the data M times, resulting in K augmented views Avg. i (V test ), i∈[1,M.

[0078] Step 10: Extract visual features from the M enhanced views obtained in Step 9 using a pre-trained video encoder:

[0079]

[0080] in, This represents the visual features of the i-th augmented view of the test video.

[0081] Step 11: Compare the visual features of the enhanced test video view obtained in Step 10 with the visual proxy Z obtained in Step 8. v Calculate the probability distribution p predicted by the visual agent for each augmented view of the test video. vp (Y|Aug i (V test )):

[0082]

[0083] Where Softmax(·) represents the Softmax function operation, τ v This is the temperature parameter used by the visual agent during Softmax calculations, and β is a parameter that adjusts the "sharpness". Support set tags for one-hot encoding.

[0084] Step 12: Compare the visual features of the enhanced test video view obtained in Step 10 with the text proxy Z obtained in Step 1. t Calculate the probability distribution p predicted by the text agent for each augmented view of the test video. tp (Y|Aug i (V test )):

[0085]

[0086] Where τ tThis is the temperature parameter used by the text agent when performing Softmax calculations.

[0087] Step 13: For each augmented view of the calculated test video, predict the probability distribution p using the visual agent. vp (Y|Aug i (V test )) and the probability distribution p predicted using a text proxy tp (Y|Aug i (V test This method utilizes the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution to dynamically adjust the learnable visual cues p for the test video during testing. v and text prompt p t :

[0088]

[0089] Among them, D KL (·) represents the KL divergence calculation between two probabilistic texts.

[0090] Step 14: Obtain the fine-tuned learnable visual cues by minimizing the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution. Obtain the final visual agent used for prediction

[0091]

[0092] in, Indicates in The visual feature of the i-th supporting video is extracted with the assistance of [unclear].

[0093] Step 15: For the visual agent obtained in step 14 used for prediction Calculate the predicted score of the test video This allows for zero-shot action recognition, i.e.:

[0094]

[0095] Table 1 compares the results of our method with other zero-shot video classification methods on four video classification datasets: HMDB-51, UCF-101, Kinetics-600, and Activity Net, with TOP-1 accuracy as the evaluation metric.

[0096] Obviously, those skilled in the art should understand that the steps of the zero-shot video classification method based on test-time visual agent tuning in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using device-executable program code, thereby storing them in a storage device for execution by the computing device. Furthermore, in some cases, the steps shown or described can be performed in a different order than presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any specific hardware and software combination.

[0097] Table 1 Comparison of results across the four datasets

[0098]

[0099] The embodiments described above do not constitute a limitation on the scope of protection of this application.

Claims

1. A zero-shot video classification method based on test-time visual agent tuning, characterized in that, The method includes: Step 1: For a test video and a given set of action categories, combine the learnable text prompts with each action category to form a category description text, and use a pre-trained visual language model to extract the visual features of the test video and the text features of the category description text respectively. Step 2: For a given set of action categories, generate multiple different action descriptions for each action category using a Large Language Model (LLM); then use the generated action descriptions to generate a batch of videos for each action category using a Text-to-Video (T2V) model; next, use a pre-trained visual language model to calculate the score of the real category to which the generated videos belong, and select the K videos with the highest scores for each action category to construct a support set. Step 3: Pass the support set into the visual agent building module to obtain the visual agent for prediction; Step 4: Utilize the bimodal cue collaborative tuning module to simultaneously adjust the learnable text and visual cues, thereby optimizing the visual agent for different test videos; Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification; Step 1 includes: For input video V test The T sampled images are used to extract corresponding visual features of dimension d using a pre-trained visual language model video encoder. Where E v It is a video encoder based on a pre-trained visual language model; For an action category set Y = {y0, y1, ..., y} with C categories, c ,}, will provide learnable text prompts p t Each action category is combined with a category description text, and the corresponding text features of dimension d are extracted using a pre-trained visual language model's text encoder as text proxies. Where E t It is a text encoder for a pre-trained visual language model; The execution process of the visual proxy building module is as follows: Step 301: For each supporting video in the support set S, sample T. S Zhang images, which will provide learnable visual cues p v Add it to the sampled image to obtain the supported set of video images; Step 302: Extract visual features from the sampled support set video images using a pre-trained video encoder and stitch them together to form a visual proxy. Where F i v represents the visual feature of the i-th supporting video. i This represents the i-th supported video in the support set. Concat(·) represents the vector concatenation operation. C and K represent the number of categories and the number of support set samples for each category, respectively. d is the feature dimension. The execution process of the bimodal feedback collaborative tuning module is as follows: Step 401: For each test video V test Aug is augmented M times using the AugMix data augmentation method, resulting in M ​​augmented views Aug. i (V test ), i∈[1,M]; Step 402: Extract visual features from the M enhanced views obtained in step 401 using a pre-trained video encoder. in, This represents the visual features of the i-th augmented view of the test video; Step 403: For the visual features of the augmented view of the test video obtained in step 402 and the visual agent Zv obtained in step 302, calculate the probability distribution p predicted by the visual agent for each augmented view of the test video. vp (Y|Aug i (V test )): Where Softmax(·) represents the Softmax function operation, τ v β is the temperature parameter used by the visual agent during Softmax calculations, and β is a parameter that adjusts sharpness. Support set tags for one-hot encoding; Step 404: Compare the visual features of the enhanced test video view obtained in step 402 with the text proxy Z obtained in step 1. t Calculate the probability distribution p predicted by the text agent for each augmented view of the test video. tp (Y|Aug i (V test )): Where τ t This is the temperature parameter used by the text agent when performing Softmax calculations; Step 405: For each augmented view of the calculated test video, predict the probability distribution p using the visual agent. vp (Y|Aug i (V test )) and the probability distribution p predicted using a text proxy tp (Y|Aug i (V test This method utilizes the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution to dynamically adjust the learnable visual cues p for the test video during testing. v and text prompt p t : Among them, D KL (·) represents the KL divergence calculation between two probabilistic texts; Step 406: Obtain the fine-tuned learnable visual cue by minimizing the KL divergence between the visual agent's prediction probability distribution and the text agent's prediction probability distribution. Obtain the final visual agent used for prediction in, Indicates in The visual feature of the i-th supporting video is extracted with the assistance of [unclear].

2. The method according to claim 1, characterized in that, The support set construction process is as follows: Step 201: For any action category y i Each action is transformed into m different action descriptions using a large language model, namely: Where LLM(·) is the Large Language Model, This represents the j-th description of the i-th action category; Step 202: For a set of action categories Each action category is processed in step 201 to generate an action description set with C×m action descriptions. Step 203: For action category y in the action category set i Using a text-to-video generation model based on action description set D, y i The corresponding m action descriptions are used to generate k video samples; Step 204: For category y i For the corresponding k generated video samples, a pre-trained visual language model is used to calculate the category of each generated video sample in y. i Based on the probability scores, select the K video samples with the highest scores; Step 205: By generating and selecting video samples for each action category using steps 203 and 204, a support set S with C×K supporting videos is finally constructed.

3. The method according to claim 2, characterized in that, Step 5: Use the optimized visual agent for inference to achieve zero-shot video classification, including: Finally, for the visual agent obtained in step 406 for prediction Calculate the predicted score of the test video This allows for zero-shot action recognition, i.e.:

Images