Zero sample video classification method based on test time support set adjustment

Abstract

According to the zero sample video classification method based on test time support set adjustment provided by the invention, zero sample action recognition of video actions is realized through a multi-prompt support set expansion module and a time perception support set erosion module, and the semantic difference problem between a vision mode and a text mode in a video classification task is relieved. And the multiple prompt support set expansion module and the time perception support set erosion module form a zero sample learning framework TEST-V. In the multi-prompt support set expansion module, a plurality of descriptions are utilized to generate videos through a text-to-video model to obtain support samples of each class, and abundant visual information is provided for subsequent zero sample reasoning. In the time perception support set erosion module, through supervision of prediction consistency, decomposition learnable weights with different time scale features are adjusted in multiple steps, the generalization ability of zero sample video classification figures is enhanced, and the modal gap between video and text modals is relieved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a zero-sample video classification method based on test time support set adjustment. Background Art

[0002] In the past few decades, action recognition research has made rapid progress and has been successfully applied to many fields, including intelligent robots, security monitoring and autonomous driving. However, the number of actions that can be recognized by existing methods is still limited and difficult to meet the growing needs of practical applications. In recent years, the powerful ability of large-scale pre-trained visual language models (VLMs) in zero-shot generalization has promoted the rapid development of zero-shot / general action recognition. Visual language models are the hope of breaking through the perceptual boundaries of existing action recognition technology. Its goal is to project the test video and category name into a joint feature space based on the pre-trained visual language model, and then assign labels to the video according to their feature similarity, thereby classifying the video.

[0003] Although the amount of pre-training data is huge, the problem of uneven feature distribution cannot be completely eliminated in the test environment. Therefore, it is necessary to further adjust the parameters or prompts of the pre-trained model from the perspective of different modalities. Existing solutions can be roughly divided into the following two categories: training-based methods and non-training methods. In training-based methods, by leveraging a large amount of video data, fine-tuning some parameters of the pre-trained model or additional learnable prompts often requires high training costs. On the other hand, non-training methods attach learnable prompts to the visual / text input sequence and adjust the prompts in a self-supervised manner for each test video at test time. In addition, some recent methods build video support sets for unseen category names by retrieval or generation (i.e., converting class names to videos) and perform intra-modal (video-to-video) alignment. However, non-training methods based on test-time tuning freeze the visual space and only adjust the text space, which cannot narrow the modality gap. In addition, support set-based methods narrow the gap through inter-modality alignment, but the support samples are fixed and cannot be dynamically adjusted according to the video.

[0004] Therefore, how to eliminate the semantic gap between modalities and better dynamically tune the support set to effectively identify unknown categories is a key challenge for zero-shot action recognition. Summary of the invention

[0005] The present application provides a zero-shot video classification method based on test time support set adjustment, which can be used to solve the technical problem of semantic distribution differences between modalities in zero-shot video classification.

[0006] The present application provides a zero-shot video classification method based on test time support set adjustment, the method comprising:

[0007] Step 1: For a test video and a given set of action categories, use the pre-trained visual language model to extract visual and text features respectively;

[0008] Step 2: For a given set of action categories, in the multi-prompt support set expansion module, a large language model (LLM) is used to generate multiple different action descriptions for each action category; then, the generated action descriptions are used to generate support videos for building the support set using the text-to-video generation model (T2V);

[0009] Step 3: Use the support set extraction features obtained in step 2 to multiply the decomposed learnable weights in the time-aware support set erosion module element by element to obtain the weighted frame-level and patch-level support set features. Then, use the method of minimizing the prediction consistency to adjust the learnable weights. Finally, use zero-shot reasoning to screen the support set for different test videos and narrow the semantic gap between different modalities of zero-shot tasks.

[0010] Step 4: Use the multi-scale optimized support set features for reasoning to achieve zero-shot video classification.

[0011] Furthermore, in step 1, for a test video and a given set of action categories, a pre-trained visual language model is used to extract visual and text features respectively, including:

[0012] For the input video V test The sampled T images use the visual encoder of the pre-trained visual language model to extract the visual features with the corresponding feature dimension d

[0013]

[0014] Among them, F vis A visual encoder for pre-training visual language models;

[0015] For C category names Y = {y 0 ,y 1 ,…,y c ,}, use the text encoder of the pre-trained visual language model to extract text features with the corresponding feature dimension d

[0016]

[0017] Among them, F txt Text encoder for pre-trained visual language models.

[0018] Furthermore, the execution process of the multiple prompt support set expansion module is as follows:

[0019] Step 201: for any action category y i , each action is converted into M different action description prompts through a large language model, namely:

[0020] d i =LLM(query_template(y i ,M))

[0021] Where query_template(·) is a carefully crafted text template used to encapsulate each action class into a text query input into LLM to obtain action descriptions for multiple scenes, characters, and perspectives. LLM(·) is a large language model;

[0022] Step 202: for an action category set Each action category is subjected to step 201 to obtain multiple action description prompts, and finally generates C×M different action description prompt sets.

[0023] Step 203, for each action category in the action category set, using the text-to-video generation model to store each category in the action description prompt set i The M different action descriptions are generated into K corresponding video samples:

[0024]

[0025] Among them, T2V(·) is the text-to-video generation model, N p is the number of videos generated according to each prompt. For ease of implementation, K is set to be divisible by T. Sample(·,·) is used to generate the video from d i A description prompt for extracting a specific number of K / T;

[0026] Step 204, by generating video samples for each action category using step 203, finally constructing a video support set containing K supporting videos for each action

[0027] Furthermore, the execution process of the time-aware support set erosion module is as follows:

[0028] Step 301, for the video support set constructed in step 204, use the visual encoder of the pre-trained visual language model to extract visual features and connect them:

[0029]

[0030] Where C, K, T and d represent the number of action categories, the number of support set samples corresponding to each action category, the number of frames and the number of channels respectively; Concat(·) means concatenating each extracted video feature one by one to obtain the extracted support set feature matrix F;

[0031] Step 302: For the support set features F extracted in step 301, the support set features are re-weighted from the video level and frame level through the decomposed learnable weights in the time-aware support set erosion module, so as to screen the support set feature matrix F from the video level and frame level:

[0032]

[0033] Among them, ⊙ is the element-by-element matrix multiplication, namely the Hadamard product, are video-level learnable weights, is the frame-level learnable weight, and F' is the re-weighted support set feature;

[0034] Step 303: For each test video feature V test Use the spatial enhancement method to enhance N times:

[0035]

[0036] in, Represents the test video feature V test Perform N times of spatial enhancement; is the enhanced test video feature;

[0037] Step 304: For the test video features enhanced in step 303 and the support set features re-weighted in step 302, calculate their multi-view prediction score Z:

[0038]

[0039] Among them, fW T and FW T are the similarities of the test video features, support set features, and category text features, respectively; KL(·) means calculating the KL divergence between two similarities;

[0040] Step 305: For the calculated multi-view prediction score Z, a method of minimizing prediction consistency is used to dynamically adjust the test video during testing:

[0041]

[0042] Wherein, Pred(·) represents the multi-view prediction score Z calculated from the weighted support set features and the enhanced test video features according to step 304, represents the average entropy calculated from high confidence predictions;

[0043] Step 306: In order to obtain more useful information at the temporal scale of the video, the weighted support set features and the enhanced test video features are sampled m times at different temporal scales to perform multi-scale temporal fine-tuning:

[0044]

[0045] in, It means that two feature sets are enhanced synchronously by sampling m times at different time scales;

[0046] Step 307: Use the minimized prediction consistency to obtain the fine-tuned learnable weights and obtain the final support set features for prediction:

[0047]

[0048] in and are the fine-tuned video-level and frame-level learnable weights, respectively, * is the support set feature used for prediction.

[0049] Furthermore, we use the multi-scale optimized support set features for reasoning to achieve zero-shot video classification, including:

[0050] For the support set feature F obtained in step 307 for prediction * , calculate the prediction score Y for zero-shot action recognition:

[0051] Y=ψ(-KL({fW T ||F * W T ))L.

[0052] The present application also provides a control server, comprising:

[0053] one or more processors;

[0054] A storage device for storing one or more programs;

[0055] When the one or more programs are executed by the one or more processors, the one or more processors implement the zero-sample video classification method based on test time support set adjustment provided by the present application.

[0056] A storage medium stores a computer program, which, when executed by a processor, implements the zero-sample video classification method based on test time support set adjustment provided by the present application.

[0057] The zero-shot video classification method based on test time support set adjustment proposed in the present invention has achieved state-of-the-art results on four datasets: HMDB-51, UCF-101, Kinetics-600, and ActiviyNet. The method provided in this application realizes advanced video classification without the need for additional training data through a multiple-cue support set expansion module and a time-aware support set erosion module, and solves the semantic gap problem between the two modalities of vision and text in zero-shot tasks. Through the language template designed in the multiple-cue support set expansion module, action descriptions with rich semantic information are obtained from advanced large language models. At the same time, a specific video generation strategy and an advanced text-to-video generation model are used to construct an action-rich and diverse video support set, thereby effectively assisting subsequent zero-shot reasoning and significantly improving the performance of traditional methods based on text-to-image generation models.

[0058] In addition, in order to solve the problem of difficult samples in the support set constructed by existing methods and the inability to optimize for specific test video data, a new self-supervised fine-tuning mechanism is introduced in the time-aware support set erosion module. During the test process, the learnable weights with different time scale characteristics are adjusted in multiple steps by predicting consistency. Then, the fine-tuned learnable weights are used to optimize the weights of the constructed support set in multiple dimensions, so as to extract key action clues from the support set for auxiliary reasoning. Finally, the state-of-the-art performance is achieved in the zero-shot video classification task. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A flowchart provided for an embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0061] The following first introduces the embodiments of the present application in conjunction with the accompanying drawings.

[0062] In order to solve the semantic gap problem between the visual and text modalities in the zero-shot video classification task, the present invention uses a zero-shot learning framework TEST-V based on the test time support set adjustment, uses an advanced large language model to generate diversified action descriptions for each action category, and uses the action descriptions to generate a video support set through a text-to-video generation model, and uses learnable weights to build a unique video support set for each test video. At the same time, multi-scale weight fine-tuning is achieved by minimizing the prediction consistency method. The modal gap between the video and text modalities is alleviated, and the model's recognition and generalization capabilities for zero-shot actions are enhanced.

[0063] like Figure 1 As shown in the figure, a zero-shot video classification method based on test time support set adjustment realizes zero-shot action recognition of video actions through multiple prompt support set expansion modules and time-aware support set erosion modules, alleviating the semantic gap problem between the visual and text modalities in video classification tasks. The multiple prompt support set expansion module and the time-aware support set erosion module constitute the zero-shot learning framework TEST-V. In the multiple prompt support set expansion module, multiple descriptions are used to generate videos through the text-to-video model to obtain support samples for each class, providing rich visual information for subsequent zero-shot reasoning. In the time-aware support set erosion module, the decomposed learnable weights with different time scale features are adjusted in multiple steps through the supervision of prediction consistency, thereby enhancing the generalization ability of zero-shot video classification of characters and alleviating the modal gap between video and text modalities. Specifically, it includes the following steps:

[0064] Step 1: For the input video V test The sampled T images use the visual encoder of the pre-trained visual language model to extract the visual features of the corresponding feature dimension d

[0065]

[0066] Among them, F vis Visual encoder for pre-trained visual language model.

[0067] For C category names Y = {y 0 ,y 1 ,…,y c ,}, use the text encoder of the pre-trained visual language model to extract the text features with the corresponding feature dimension d

[0068]

[0069] Among them, F txt Text encoder for pre-trained visual language models.

[0070] Step 2: For any action category y i , each action is converted into M different action description prompts through a large language model, namely:

[0071] d i =LLM(query_template(y i ,M))

[0072] Where query_template(·) is a carefully crafted text template used to encapsulate each action class into a text query input into LLM to obtain action descriptions for multiple scenes, characters, and perspectives. LLM(·) is a large language model.

[0073] Step 3: For an action category set Each action category is processed through step 201, and finally C×M different action description prompt sets are generated.

[0074] Step 4: For each action category in the action category set, use the text-to-video generation model to store each category in the action description prompt set d i The M different action descriptions are generated into K corresponding video samples:

[0075]

[0076] Among them, T2V(·) is the text-to-video generation model, N p is the number of videos generated for each prompt. For ease of implementation, we always set K to be divisible by T. Sample(·,·) is used to generate i A description prompt for extracting a specific number of K / T.

[0077] Step 5: Generate video samples for each action category using step 203, and finally construct a video support set containing K support videos for each action

[0078] Step 6: For the video support set constructed in step 204, use the visual encoder of the advanced pre-trained visual language model to extract its visual features and connect them:

[0079]

[0080] Where C, K, T and d represent the number of action categories, the number of support set samples corresponding to each action category, the number of frames and the number of channels respectively. Concat(·) means concatenating each extracted video feature one by one to obtain the extracted support set feature matrix F.

[0081] Step 7: For the support set features F extracted in step 301, the support set features are re-weighted from two dimensions, video level and frame level, through the decomposed learnable weights in the time-aware support set erosion module:

[0082]

[0083] Among them, ⊙ is the element-by-element matrix multiplication, namely the Hadamard product, are video-level learnable weights, is the frame-level learnable weight, and F' is the re-weighted support set feature.

[0084] Step 8: For each test video feature V test We use the spatial enhancement method to enhance it N times:

[0085]

[0086] in, Represents the test video feature V test Perform N times of spatial enhancement. It is the enhanced test video feature.

[0087] Step 9: For the test video features enhanced in step 303 and the support set features re-weighted in step 302, calculate the multi-view prediction score Z:

[0088]

[0089] Among them, fW T and FW T are the similarities of the test video features, support set features, and category text features, respectively. KL(·) means calculating the KL divergence between two similarities.

[0090] Step 10: For the calculated multi-view prediction score Z, the test video is dynamically adjusted during testing by minimizing the prediction consistency:

[0091]

[0092] Wherein, Pred(·) represents the multi-view prediction score Z calculated from the weighted support set features and the enhanced test video features according to step 304, Denotes the average entropy calculated from high confidence predictions.

[0093] Step 11: In order to obtain more useful information at the temporal scale of the video, the weighted support set features and enhanced test video features are sampled m times at different temporal scales to perform multi-scale temporal fine-tuning:

[0094]

[0095] in, It means that two feature sets are synchronously enhanced by sampling m times at different time scales.

[0096] Step 12: Use the minimized prediction consistency to obtain the fine-tuned learnable weights and obtain the final support set features for prediction:

[0097]

[0098] in and are the fine-tuned video-level and frame-level learnable weights, respectively, * is the support set feature used for prediction.

[0099] Step 13: For the support set features F obtained in step 307 for prediction * , calculate its prediction score Y, and then perform zero-sample action recognition:

[0100] Y=ψ(-KL({fW T ||F * W T ))L

[0101]

[0102]

[0103] Table 1 Comparison of results on four datasets

[0104] Table 1 compares the results of this method with other methods on four video classification datasets: HMDB-51, UCF-101, Kinetics-600, and ActivityNet, with TOP-1 accuracy as the evaluation indicator.

[0105] Obviously, those skilled in the art should understand that the various steps of the zero-sample video classification method based on test time support set adjustment of the above-mentioned embodiment of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented with program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to being combined with any specific hardware and software

[0106] The above-described embodiments of the present application do not constitute a limitation on the protection scope of the present application.

Claims

1. A zero-shot video classification method based on test time support set adjustment, characterized in that: The method comprises: Step 1: For a test video and a given set of action categories, use the pre-trained visual language model to extract visual and text features respectively; Step 2: for a set of action categories given in step 1, in the multi-prompt support set expansion module, a large language model (LLM) is used to generate multiple different action descriptions for each action category; then, the generated action descriptions are used to generate support videos for constructing the support set using the text-to-video generation model (T2V); Step 3: Use the support set extraction features obtained in step 2 to multiply the decomposed learnable weights in the time-aware support set erosion module element by element to obtain weighted frame-level and patch-level support set features. Then, use the method of minimizing prediction consistency to adjust the learnable weights. Finally, use zero-sample inference to screen the support set for different test videos. Step 4: Use the multi-scale optimized support set features for reasoning to achieve zero-shot video classification.

2. The method according to claim 1, characterized in that: Step 1: For a test video and a given set of action categories, use the pre-trained visual language model to extract visual and text features, including: For the input video V test The sampled T images use the visual encoder of the pre-trained visual language model to extract the visual features with the corresponding feature dimension d Among them, F vis A visual encoder for pre-training visual language models; For C category names Y = {y0, y1, ..., y c ,}, use the text encoder of the pre-trained visual language model to extract text features with the corresponding feature dimension d where F txt Text encoder for pre-trained visual language models.

3. The method according to claim 2, characterized in that The execution process of the multiple prompt support set expansion module is as follows: Step 201: for any action category y i , each action is converted into M different action description prompts through a large language model, namely: d i =LLM(query_template(y i ,M)) Where query_template(·) is a text template used to encapsulate each action class into a text query input into LLM to obtain action descriptions of multiple scenes, characters, and perspectives. LLM(·) is a large language model; Step 202: for an action category set Each action category is subjected to step 201 to obtain multiple action description prompts, and finally generates C×M different action description prompt sets. Step 203, for each action category in the action category set, using the text-to-video generation model to store each category in the action description prompt set i The M different action descriptions are generated into K corresponding video samples: Among them, T2V(·) is the text-to-video generation model, N p is the number of videos generated for each prompt, K is set to be divisible by T, and Sample(·,·) is used to generate i A description prompt for extracting a specific number of K / T; Step 204, by generating video samples for each action category using step 203, finally constructing a video support set containing K supporting videos for each action 4. The method according to claim 3, characterized in that The execution process of the time-aware support set erosion module is as follows: Step 301, for the video support set constructed in step 204, use the visual encoder of the pre-trained visual language model to extract visual features and connect them: Where C, K, T and d represent the number of action categories, the number of support set samples corresponding to each action category, the number of frames and the number of channels respectively; Concat(·) means concatenating each extracted video feature one by one to obtain the extracted support set feature matrix F; Step 302: For the support set features F extracted in step 301, the support set features are re-weighted from the video level and frame level through the decomposed learnable weights in the time-aware support set erosion module, so as to screen the support set feature matrix F from the video level and frame level: Among them, ⊙ is the element-by-element matrix multiplication, namely the Hadamard product, are video-level learnable weights, is the frame-level learnable weight, and F' is the re-weighted support set feature; Step 303: For each test video feature V test Use the spatial enhancement method to enhance N times: in, Represents the test video feature V test Perform N times of spatial enhancement; is the enhanced test video feature; Step 304: For the test video features enhanced in step 303 and the support set features re-weighted in step 302, calculate their multi-view prediction score Z: Among them, fW T and FW T are the similarities of the test video features, support set features, and category text features, respectively; KL(·) means calculating the KL divergence between two similarities; Step 305: For the calculated multi-view prediction score Z, a method of minimizing prediction consistency is used to dynamically adjust the test video during testing: Where Pred(·) represents the multi-view prediction score Z calculated from the weighted support set features and the enhanced test video features according to step 304, represents the average entropy calculated from high confidence predictions; Step 306: Sampling the weighted support set features and enhanced test video features m times at different temporal scales to perform multi-scale temporal fine-tuning: in, It means that two feature sets are enhanced synchronously by sampling m times at different time scales; Step 307: Use the minimized prediction consistency to obtain the fine-tuned learnable weights and obtain the final support set features for prediction: in and are the fine-tuned video-level and frame-level learnable weights, respectively, * is the support set feature used for prediction.

5. The method according to claim 4, characterized in that Use multi-scale optimized support set features for reasoning to achieve zero-shot video classification, including: For the support set feature F obtained in step 307 for prediction * , calculate the prediction score Y for zero-shot action recognition: Y=ψ(-KL({fW T |F * W T ))L。 6. A control server, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the zero-sample video classification method based on test time support set adjustment as described in any one of claims 1-5.

7. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the zero-sample video classification method based on test time support set adjustment as described in any one of claims 1 to 5 is implemented.

Images