Video-to-text discrete video identification method and device based on text guidance

Abstract

The invention discloses a video-to-text discrete video identification method and device based on text guidance, and the method comprises the steps: employing a CLIP text and a visual encoder, and extracting feature information from a text frame and a video frame; adding learnable prompt in the feature extraction process, performing fine adjustment on the features to generate learnable text features, and performing dynamic dictionary construction by using the learnable text features; obtaining a dynamic dictionary and video features, and obtaining discrete features corresponding to a video through a video-to-text discrete module; and inputting the discrete features and the video frame features into a confidence fusion module together, performing preliminary fusion on the discrete features and the video frame features to obtain video frame features aggregated with the discrete features, and further fusing the video frame features through confidence scores to obtain feature expression of the whole video. The device comprises a processor and a memory. According to the method, fusion of the video frame features and the discrete text features is realized, so that more accurate video feature description is obtained.

Description

Technical Field

[0001] The present invention relates to the fields of multimodal image processing and video recognition, and in particular to a video recognition method and device based on text-guided video-to-text discreteness. Background Art

[0002] Large-scale visual language models have made significant progress in the field of artificial intelligence after being pre-trained on a large number of carefully curated image-text pair datasets, successfully bridging the gap between vision and language, thereby improving the machine's ability to understand visual data and generate descriptions that are similar to human language content. Visual language models like CLIP have a wide range of applications in multiple tasks, including: image caption generation, visual question answering, image-text retrieval, and multimodal generation. Given the success of these models in previous tasks, researchers hope to extend these pre-trained visual language models to the field of video recognition. However, fully fine-tuning the model may lead to catastrophic forgetting. Therefore, researchers are exploring parameter-efficient fine-tuning methods as an ideal alternative to adapting pre-trained CLIP models for video recognition.

[0003] Videos consisting of highly correlated frames can be effectively analyzed using pre-trained CLIP models. In recent years, many methods for video recognition have emerged. These methods can be roughly divided into two categories: prompt-based methods and Adapter-based methods. Prompt-based methods use learnable visual and text prompts to achieve video-text alignment. Prompt-based methods perform well on benchmark datasets, making prompt learning a widely used technique for efficient parameter fine-tuning. However, due to the limited number of parameters, they are only applicable to simple downstream tasks and cannot cope with more complex challenges such as video recognition. Adapter-based methods freeze the CLIP backbone network and add trainable layers on top of frame features for temporal modeling.

[0004] Previous CLIP-based methods were used to capture temporal dependencies across frames. However, it is unclear whether the frame features from CLIP contain temporal or similarity relationships, and the temporal modeling capabilities of the temporal module have not yet been verified. In addition, these methods use independent encoders for video-text contrastive learning, but ignore text representations in video modeling, limiting their generalization capabilities. How to better utilize text features and how to better construct temporal modeling modules are the research contents of many works in recent years. Summary of the invention

[0005] The present invention provides a video recognition method and device based on text-guided video-to-text discretization. The present invention designs a feature extraction module based on the CLIP large model, uses a learnable prompt to extract video and text features, and uses text features to build a dynamic dictionary; through similarity calculation and a dynamic voting mechanism between video frames and texts, the video is discretized into texts; through a confidence feature fusion strategy, the fusion of video frame features and discrete text features is achieved, thereby obtaining a more accurate video feature description, as described below:

[0006] In a first aspect, a video recognition method based on text-guided video to text discreteness is provided, the method comprising:

[0007] Use CLIP's text and visual encoders to extract feature information from text and video frames respectively;

[0008] Add a learnable prompt in the feature extraction process, fine-tune the features to generate learnable text features, and use the learnable text features to construct a dynamic dictionary;

[0009] Get the dynamic dictionary and video features, and use the video-to-text discrete module to get the discrete features corresponding to the video;

[0010] The discrete features and video frame features are input into the confidence fusion module together, and the discrete features and video frame features are initially fused to obtain the video frame features that aggregate the discrete features. The video frame features are further fused through the confidence score to obtain the feature expression of the entire video.

[0011] Among them, the method uses the similarity between frames and texts to obtain the text with the maximum similarity corresponding to each frame, and the texts corresponding to all video frames constitute candidate texts; then frame-level voting is performed, and similarity is used as the voting weight. The text feature with the highest vote score is counted as the discrete feature corresponding to the video.

[0012] The process of adding a learnable prompt in the feature extraction process, fine-tuning the features to generate learnable text features, and constructing a dynamic dictionary using the learnable text features is specifically as follows:

[0013] Text encoder φ using CLIP t (·;θ t ) to extract text features; use the encoder parameters pre-trained by CLIP for initialization. For category text, add a learnable prompt before entering the text encoder;

[0014] Add a randomly initialized learnable prompt before the text input to the text encoder to generate text features. The input y of the text encoderk for:

[0015] y k =[w1][w2]…[w n ][class k ],k∈[1,K]

[0016] Among them, K is the number of categories, w i is a learnable vector, and the text feature is c i for:

[0017] c i =φ t (y k θ t )

[0018] Among them, θ t is the text encoder parameter;

[0019] Constructing a dynamic dictionary using learnable text features extracted by a text encoder

[0020] The discrete features corresponding to the video are obtained by the discrete module from video to text:

[0021] Visual encoder φ using CLIP v (·;θ v ) to extract video frame features; use the encoder parameters pre-trained by CLIP to initialize, for an input video V = {I t}, the video is evenly divided into T segments, and a random frame is selected from each segment to obtain T frames. The sampled video frames are input into the visual encoder in sequence. When inputting, each video frame is divided into A patch of size p×p is added before each frame with a randomly initialized learnable prompt, which is then input into the visual encoder to obtain the video frame features. The input s of the visual encoder is t for:

[0022] s t =[u1][u2]…[u m ][e t ],t∈[1,T]

[0023] Among them, T is the number of frames of each video, u i is a learnable vector, and the video feature x is obtained t for:

[0024] x t =φ v (s t θ v )

[0025] Among them, θ v is the visual encoder parameter;

[0026] Use a dynamic dictionary to measure the features of video frames. The measurement method is:

[0027]

[0028] in, It can be used as a discrete feature.

[0029] Among them, the candidate text is:

[0030] [prd,ind]=argmax(S,0)

[0031] Where prd represents the maximum confidence score of each frame, ind represents the corresponding index in the dictionary to which each frame should be assigned, and the index is used to obtain the candidate text features. The mask matrix M = one-hot (S :,ind[:] ) The rough classification results of all frames are counted, and all the obtained texts constitute the candidate texts.

[0032] The discrete features and the video frame features are input into the confidence fusion module together, and the discrete features and the video frame features are preliminarily fused to obtain the video frame features that aggregate the discrete features. Specifically, the discrete features are:

[0033] The video discrete features and video frame features are initially fused through the cross attention mechanism, and then concat operation is performed with the original features to obtain the fused video frame features f containing discrete features t :

[0034] f t =Concat(CrossAttention(D,x t ,x t ),x t )

[0035] Then the similarity between the original video frame and the discrete features is calculated and normalized to obtain the weight w of the video frame. t :

[0036] s t =sim(x t ,D),t∈[1,T]

[0037]

[0038] Where τ is a temperature parameter, s t represents the similarity between the t-th frame of the video and the discrete text, w t Represents the correlation between the t-th video frame and the discrete text after similarity normalization;

[0039] Adopt the confidence fusion strategy and use w t The value of is used as the confidence score for the fused video frame feature f t Weighted, and finally generate the final features of the video by summing

[0040]

[0041] In a second aspect, a text-guided video-to-text discrete video recognition device is provided, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to enable the device to execute any one of the methods described in the first aspect.

[0042] In a third aspect, a computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes any one of the methods described in the first aspect.

[0043] The beneficial effects of the technical solution provided by the present invention are:

[0044] 1. The present invention proposes a text-guided video-to-text discretization method, which can discretize video features into text, thereby utilizing the stability of text features and the accuracy of expression, improving the model's expression of video features, and enhancing its generalization ability;

[0045] 2. The present invention designs a dynamic dictionary based on a text encoder, and uses the learnable text features obtained by the encoder to construct a dynamic dictionary, which reduces the difficulty of building a dynamic dictionary from scratch, reduces the amount of training, and avoids possible conflicts or collapse problems in the future;

[0046] 3. The present invention proposes a feature fusion method based on confidence scores, which preliminarily fuses discrete text features and can integrate the advantages of text features into video features. In addition, the confidence score fusion method based on similarity calculation gives different confidences to video frames, which can effectively reduce the interference of video redundancy or noise information to video recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of the overall network structure;

[0048] Figure 2 Flowchart of a video recognition method based on text-guided video to text discretization. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.

[0050] In order to solve the problems existing in the background technology, the embodiment of the present invention proposes a text-guided video-to-text discretization video recognition method, which can fuse the stability of text features and their clear semantic information into the features of the video, thereby improving the model's expression of video features and improving the model's ability to recognize videos. The network includes: a feature extraction module, a video-to-text discretization module, and a confidence fusion module. The video recognition method avoids the uncertainty of temporal modeling and can better utilize text features to improve video recognition performance.

[0051] Example 1

[0052] An embodiment of the present invention provides a video recognition method based on text guidance to discretize text from video, the method comprising the following steps:

[0053] 101: The feature extraction module uses CLIP's text and visual encoders, which use powerful multimodal learning capabilities to extract rich feature information from text and video frames respectively; in addition, this module is initialized using CLIP's pre-trained parameters;

[0054] Among them, the above steps enable the feature extraction process to not only have good visual and language feature alignment capabilities, but also make full use of the generalization ability learned by the pre-trained model on large-scale data.

[0055] 102: Add a learnable prompt in the feature extraction process, fine-tune the features, add a learnable prompt to the first 9 layers of the encoder, generate learnable text features, make the text and visual features more aligned, and use the learnable text features to construct a dynamic dictionary;

[0056] 103: Obtain a dynamic dictionary and video features, and obtain discrete features corresponding to the video through a video-to-text discrete module;

[0057] After obtaining the dictionary and video frame features, the similarity between the frame and the text is used to first obtain the text with the maximum similarity corresponding to each frame. The texts corresponding to all video frames constitute the candidate texts. After obtaining the candidate texts, frame-level voting is performed, and the similarity is used as the voting weight. The text feature with the highest vote score is the discrete feature corresponding to the video.

[0058] 104: Obtain discrete features, input them together with video frame features into a confidence fusion module, preliminarily fuse the discrete features with the video frame features to obtain video frame features that aggregate the discrete features, and then further fuse the video frame features through confidence scores to obtain feature expressions of the entire video.

[0059] In summary, this method can fuse the stability of text features and their clear semantic information into the features of the video through a video recognition model consisting of three modules: feature extraction module, video to text discretization module and confidence fusion module, thereby improving the model's ability to express video features; this video recognition method avoids the uncertainty of temporal modeling and can better utilize text features to improve video recognition performance.

[0060] Example 2

[0061] The scheme in Example 1 is further introduced below in combination with specific examples and calculation formulas, as described below for details:

[0062] 1. Data Preparation

[0063] The embodiments of the present invention verify the effectiveness of the proposed method on four datasets: HMDB-51, UCF-101, SSv2 and Kinetics-400.

[0064] The HMDB51 dataset is a widely used benchmark dataset for video action recognition tasks, containing video clips from real-world scenes. The dataset consists of 51 different action categories, covering a variety of complex actions and scenes, such as dancing, running, punching, riding a bicycle, etc. Each action category contains about 100 videos, providing a total of about 6,800 video clips.

[0065] The UCF101 dataset contains 101 different action categories, which cover a variety of actions in daily life, including sports (e.g., basketball, football), daily activities (e.g., cleaning, drinking water), music performances (e.g., playing the piano, playing drums), and stunts. The dataset contains 13,320 video clips, and the video length is usually between a few seconds and tens of seconds. The action categories in the UCF101 dataset are similar (e.g., a variety of different ball sports), and the video backgrounds also vary greatly. In addition, some videos in UCF101 are of low quality, which brings additional difficulties to video recognition.

[0066] The SSv2 (Something-Something V2) dataset is a benchmark dataset specifically for video action understanding and temporal modeling tasks. Unlike traditional video datasets, the SSv2 dataset focuses more on capturing actions and their temporal relationships through short videos, and is suitable for studying the importance of temporal features for action recognition. The SSv2 dataset contains 174 different action categories, more than 220,000 video samples, each video is 2 to 6 seconds long, and each category contains a large number of videos.

[0067] Kinetics-400 contains 400 different action categories, which cover a wide range of daily activities, sports, entertainment activities, musical instrument playing, etc. For example, there are categories such as "playing football", "riding a bicycle", "playing the piano", and "dancing". These categories not only involve human actions, but also include interactions between people and objects, and people and the environment. The dataset contains more than 240,000 video clips, all of which are from YouTube and are usually about 10 seconds long. The video clips are of high quality, and the number of video samples for each action category is sufficient, usually containing 400-600 video clips for each category. The training set contains about 240,000 video clips. The validation set and test set each contain about 20,000 video clips.

[0068] 2. Video to Text Discrete to Video Recognition Model Based on Text Guidance

[0069] The text-guided video-to-text discrete-to-video recognition model in the embodiment of the present invention is as follows: Figure 1 As shown, it includes three modules: feature extraction module, video to text discretization module and confidence fusion module.

[0070] In the embodiment of the present invention, the feature extraction module uses the text and visual encoders of CLIP. These two encoders utilize powerful multimodal learning capabilities and can extract rich feature information from text and video frames respectively. In addition, the module uses the pre-trained parameters of CLIP for initialization, which enables the feature extraction process to not only have good visual and language feature alignment capabilities, but also to make full use of the generalization capabilities learned by the pre-trained model on large-scale data. In order to better generate video frames and text features, the embodiment of the present invention adds a learnable prompt to the input part of both encoders, and uses the learnable text features to build a dynamic dictionary. The obtained visual features and dynamic dictionary enter the discrete module of video to text, and through similarity calculation, the coarse classification information corresponding to the video frame is selected. The coarse classification information of all frames constitutes the candidate text, and the similarity score between the frame and the candidate text is used for voting. The candidate text feature with the highest total score is the discrete feature of the video. The discrete features and video frame features are first preliminarily fused through the attention mechanism and jump links, and then the confidence of the video frame fusion is obtained by using the similarity between the video frame and the discrete features. The video frame features are fused according to the confidence to obtain the entire video feature. Finally, the obtained video features are used for video classification and recognition, and the entire model is trained using contrastive learning.

[0071] Among them, the implementation steps of constructing a dynamic dictionary are:

[0072] Text encoder φ using CLIP t (·;θ t ) to extract text features; use the encoder parameters pre-trained by CLIP for initialization. For category text, add a learnable prompt before entering the text encoder;

[0073] In the process of text feature extraction, a randomly initialized learnable prompt is first added before the text input to the text encoder to generate text features. k for:

[0074] y k =[w1][w2]…[w n ][class k ], k∈[1,K] (1)

[0075] Among them, K is the number of categories, w i is a learnable vector, and the text feature is c i for:

[0076] c i =φ t (y k θ t ) (2)

[0077] Among them, θ t is the text encoder parameter. By adding a learnable prompt, the model can capture variable context information, making the model better adapted to different task requirements.

[0078] In the process of dynamic dictionary construction, predefined category texts can effectively represent visual clusters and reduce conflicts. Therefore, the embodiment of the present invention uses the learnable text features extracted by the text encoder to construct the dynamic dictionary. With this design, the model can not only achieve accurate feature mapping between different categories, but also automatically adjust the dictionary during training to gradually adapt it to the changing features in the video data, thereby enhancing the matching ability between video and text.

[0079] The implementation steps of the text-guided video-to-text discretization mechanism are as follows: First, use the CLIP visual encoder φ v (·;θ v ) to extract video frame features; use the encoder parameters pre-trained by CLIP to initialize, for an input video V = {I t},in The video is evenly divided into T segments, and then a frame is randomly selected from each segment to obtain T frames. The sampled video frames are then input into the visual encoder in sequence. During input, each video frame is divided into A patch of size p×p is added before each frame with a randomly initialized learnable prompt, which is then input into the visual encoder to obtain the video frame features. The input s of the visual encoder is t for:

[0080] s t =[u1][u2]…[u m ][e t ], t∈[1,T] (3)

[0081] Among them, T is the number of frames of each video, u i is a learnable vector, and the video feature x is obtained t for:

[0082] x t =φ v (s t θ v ) (4)

[0083] Among them, θ vis the visual encoder parameter. By adding these prompt words before the frame, the image encoder can not only understand each video frame from different perspectives, but also obtain additional contextual information through the prompt words, thereby enhancing the performance of the model in the video sequence.

[0084] Getting the dynamic dictionary and video frame features x t After that, the dynamic dictionary can be used to measure the features of the video frame like VQ-VAE. The measurement method is:

[0085]

[0086] in, It can be used as a discrete feature.

[0087] On this basis, the embodiment of the present invention adjusts the measurement method. First, the similarity s between the t-th video frame and the k-th text is calculated by cosine similarity. tk :

[0088] s tk =sim(x t ,c k ),t∈[1,T],k∈[1,K] (6)

[0089] Among them, sim(·) is the cosine similarity calculation, x t is the video frame feature, c k is the text feature, K is the number of text categories, and T is the number of video frames, which ultimately form the similarity matrix S.

[0090] Then, the video frame is roughly classified. By obtaining the similarity between the video frame and the text, the text feature with the highest similarity corresponding to each video frame is obtained as the rough classification of the frame. The calculation method is:

[0091] [prd,ind]=argmax(S,0) (7)

[0092] Among them, prd represents the maximum confidence score of each frame, ind represents the corresponding index in the dictionary to which each frame should be assigned, and the index is used to obtain the candidate text features. Then use the mask matrix M = one-hot (S :,ind[:] ) The rough classification results of all frames are counted, and all the obtained texts constitute the candidate texts.

[0093] Then, the similarity value of each frame corresponding to its coarse classification is counted as the voting result of the frame, and the final score of all candidate texts is counted. The highest text feature is the discrete feature of the video. The voting result is calculated as follows:

[0094] [pm,i]=max(sum(M⊙S,-1)) (8)

[0095] Among them, i is the subscript corresponding to the discrete text feature corresponding to the video, that is, D = c i is a discrete feature.

[0096] Among them, the implementation steps of the confidence-based feature fusion strategy are:

[0097] After obtaining the discrete features of the video, they are initially fused with the video frame features through the cross attention mechanism, and then concat operation is performed with the original features to obtain the fused video frame features f containing the discrete features t ;

[0098] f t =Concat(CrossAttention(D,x t ,x t ),x t ) (9)

[0099] Then the similarity between the original video frame and the discrete features is calculated, and then normalized to obtain the weight w of the video frame. t :

[0100] s t =sim(x t ,D),t∈[1,T] (10)

[0101]

[0102] Where τ is a temperature parameter, s t represents the similarity between the t-th frame of the video and the discrete text, w t represents the correlation between the t-th frame of the video and the discrete text after similarity normalization. The embodiment of the present invention adopts a confidence fusion strategy and uses w t The value of is used as the confidence score for the fused video frame feature f t Weighted, and finally generate the final features of the video by summing

[0103]

[0104] In contrastive training, the embodiment of the present invention maximizes the cosine similarity between the video representation and its corresponding text representation in each batch through cross entropy loss to fine-tune the CLIP model. The contrastive learning optimization goal is:

[0105]

[0106] III. Evaluation Metrics and Protocols

[0107] For all trackers, the present invention embodiment uses the success rate (Acc) and precision rate (Pre) for evaluation. For a certain category, the present invention embodiment can record the correctly detected target data as TP (True Positive), the data of non-targets mistakenly identified as targets as FP (False Positive), the data of targets mistakenly identified as non-targets as FN (False Negatives), and the correctly detected non-target data as TN (True Negatives), so as to obtain the confusion matrix as shown in Table 1.

[0108] Table 1 Confusion matrix

[0109]

[0110]

[0111] Acc represents the proportion of all correct results of the classification model to the total observations.

[0112]

[0113] Pre is the proportion of outcomes for which the model prediction is P.

[0114] 4. Details of model usage

[0115] 1. Data expansion: Due to limited computing resources, we randomly flip and crop training video frames to increase the diversity of training data. For video frames larger than 680 × 640, we first adjust the size of the video frame to less than 680 × 640. We also manually supplement annotations to improve the accuracy of annotations.

[0116] 2. Model optimization:

[0117] In this method, the batch size N is set to 64 during training, and then the Adam optimization algorithm is used to optimize the training data with an initial 8×10 -4 The network is trained with a learning rate of .

[0118] 3. Video recognition:

[0119] The feature extraction module in the embodiment of the present invention can obtain category text features and video frame features. The category text features include all text categories of the video. The obtained text features constitute a dynamic dictionary. The video frame features obtain discrete features of the video through the video-to-text discrete module, and the discrete features are fused with the video frame features. Then, the obtained features are further fused through the confidence score to obtain the feature expression of the video. The feature contains both the overall video features and the discrete text information, which improves the model's ability to express video features. The model training in the embodiment of the present invention uses a contrastive learning method.

[0120] The embodiment of the present invention has the following three key innovations:

[0121] 1. A video recognition model that discretizes video to text is proposed. Technical effect: This model discretizes video to text, integrates the stability of text features and their clear semantic information into the features of the video, thereby improving the model's ability to express video features. This video recognition model avoids the uncertainty of temporal modeling and can better utilize text features to improve video recognition performance.

[0122] 2. Propose a dynamic dictionary construction method

[0123] Technical effect: The embodiment of the present invention adopts a dynamic dictionary built based on a text encoder, and constructs a dynamic dictionary by using the learnable text features generated by the encoder. This method reduces the complexity of building a dictionary from scratch, reduces the workload required for training, and effectively avoids possible conflicts or collapse problems that may occur later.

[0124] 3. Propose a feature fusion strategy based on confidence score

[0125] Technical effect: The embodiment of the present invention can integrate the advantages of text features into video features by preliminarily fusing discrete features and video frame features, and give different confidences to video frames based on the confidence score fusion method calculated based on similarity, which can effectively reduce the interference of video redundancy or noise information to video recognition.

[0126] In summary, the embodiment of the present invention proposes a video-to-text discrete video recognition model, which converts video features into text features and makes full use of the stability and clear semantic information of text features, thereby improving the model's ability to express video features and avoiding the uncertainty of temporal modeling. At the same time, a dynamic dictionary method based on text encoder construction is proposed, which uses learnable text features to build a dynamic dictionary, simplifies the dictionary construction process, reduces training complexity, and effectively avoids potential conflicts and collapse problems. In addition, the embodiment of the present invention designs a feature fusion strategy based on confidence scores, which fuses discrete features and video frame features, and uses similarity calculation to assign different confidences to video frames, effectively reducing the impact of redundant and noise information on video recognition performance.

[0127] Example 3

[0128] The embodiment of the present invention proposes to compare with multiple video recognition methods on HMDB-51, UCF-101, SSv2 and Kinetics-400 datasets. It is compared with the video recognition methods based on large-scale image pre-training models, namely Uniformer, Swin, ViViT, TokenLearner and MViTv2; and with the video recognition methods based on pre-trained large models for fine-tuning, namely ActionCLIP, XCLIP, ViFi-CLIP, ST-Adapter, STAN-conv, Vita-CLIP and M2-CLIP.

[0129] The experimental results 1 on the Kinetics-400 dataset are shown in Table 1. All the test indicators Top-1, Top-5, Avg and GFLOPs are 85.1, 97.1, 91.1 and 194 respectively, which are obviously competitive compared with the performance of other comparison methods. In terms of the three indicators of Top-1, Top-5 and Avg, the method of the present invention is higher than other comparison methods. Therefore, it can be proved that the method of the present invention can better analyze the video content, better utilize the advantages of text features, and improve the model's ability to express and recognize video features.

[0130] The results of the few-sample experiments on HMDB-51, UCF-101 and SSv2 datasets are shown in Table 2. On the HMDB-51 dataset, the experimental index Top-1 is 57.9, 62.8, 65.6 and 68.0 when the sample numbers k=2, k=4, k=8 and k=16 respectively; on the UCF-101 dataset, the experimental index Top-1 is 82.4, 86.9, 91.3 and 93.0 when the sample numbers k=2, k=4, k=8 and k=16 respectively; on the SSv2 dataset, the experimental index Top-1 is 7.1, 8.7, 10.4 and 13.0 when the sample numbers k=2, k=4, k=8 and k=16 respectively. It can be seen that with the increase in the number of videos, the recognition accuracy of the method proposed in the present invention gradually increases, and is higher than the comparison method. It proves that the method in the present invention can achieve good results under the training condition of fewer samples, thereby further verifying the effectiveness of the method.

[0131] The experimental results 3 adopted in the embodiment of the present invention are shown in Table 3. The results show the test performance of three variants of the present method on the UCF-101 dataset. The three methods are VTD-CLIP (no), VTD-CLIP (fix), and VTD-CLIP (w / o conf). VTD-CLIP (no) means that no learnable prompt is added to the text input, VTD-CLIP (fix) means that the text input uses a fixed prompt: "a photo of a", and VTD-CLIP (w / o conf) does not use confidence fusion. All variants are trained on the training set and tested on the test set. In different experiments, the training steps and other parameters are the same, while the evaluation is the same. As shown in Table 3, the present method achieves better results than its variants, and at the same time verifies that the dynamic dictionary and confidence fusion can significantly improve the model's expression of video features, thereby improving the performance of the model in video recognition tasks.

[0132] Table 1

[0133]

[0134] Table 2

[0135]

[0136] Table 3

[0137]

[0138]

[0139] Example 4

[0140] A video recognition device based on text guidance to discretize text from video, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to make the device execute the following method steps in embodiment 1:

[0141] Use CLIP's text and visual encoders to extract feature information from text and video frames respectively;

[0142] Add a learnable prompt in the feature extraction process, fine-tune the features to generate learnable text features, and use the learnable text features to construct a dynamic dictionary;

[0143] Get the dynamic dictionary and video features, and use the video-to-text discrete module to get the discrete features corresponding to the video;

[0144] The discrete features and video frame features are input into the confidence fusion module together, and the discrete features and video frame features are initially fused to obtain the video frame features that aggregate the discrete features. The video frame features are further fused through the confidence score to obtain the feature expression of the entire video.

[0145] Among them, the similarity between the frame and the text is used to obtain the text with the maximum similarity corresponding to each frame, and the text corresponding to all video frames constitutes the candidate text; then frame-level voting is performed, and the similarity is used as the voting weight. The text feature with the highest vote score is counted as the discrete feature corresponding to the video.

[0146] In the feature extraction process, a learnable prompt is added, the features are fine-tuned to generate learnable text features, and the dynamic dictionary is constructed using the learnable text features as follows:

[0147] Text encoder φ using CLIP t (·;θ t ) to extract text features; use the encoder parameters pre-trained by CLIP for initialization. For category text, add a learnable prompt before entering the text encoder;

[0148] Add a randomly initialized learnable prompt before the text input to the text encoder to generate text features. The input y of the text encoder k for:

[0149] y k =[w1][w2]…[w n ][class k ],k∈[1,K]

[0150] Among them, K is the number of categories, w i is a learnable vector, and the text feature is ci for:

[0151] c i =φ t (y k θ t )

[0152] Among them, θ t is the text encoder parameter;

[0153] Constructing a dynamic dictionary using learnable text features extracted by a text encoder

[0154] Among them, through the discrete module of video to text, the discrete features corresponding to the video are obtained as follows:

[0155] Visual encoder φ using CLIP v (·;θ v ) to extract video frame features; use the encoder parameters pre-trained by CLIP to initialize, for an input video V = {I t}, the video is evenly divided into T segments, and a random frame is selected from each segment to obtain T frames. The sampled video frames are input into the visual encoder in sequence. When inputting, each video frame is divided into A patch of size p×p is added before each frame with a randomly initialized learnable prompt, which is then input into the visual encoder to obtain the video frame features. The input s of the visual encoder is t for:

[0156] s t =[u1][u2]…[u m ][e t ],t∈[1,T]

[0157] Among them, T is the number of frames of each video, u i is a learnable vector, and the video feature x is obtained t for:

[0158] x t =φ v (s t θ v )

[0159] Among them, θ v is the visual encoder parameter;

[0160] Use a dynamic dictionary to measure the features of video frames. The measurement method is:

[0161]

[0162] in, It can be used as a discrete feature.

[0163] The candidate texts are:

[0164] [prd,ind]=argmax(S,0)

[0165] Where prd represents the maximum confidence score of each frame, ind represents the corresponding index in the dictionary to which each frame should be assigned, and the index is used to obtain the candidate text features. The mask matrix M = one-hot (S :,ind[:] ) The rough classification results of all frames are counted, and all the obtained texts constitute the candidate texts.

[0166] Among them, the discrete features and the video frame features are input into the confidence fusion module together, and the discrete features and the video frame features are preliminarily fused to obtain the video frame features that aggregate the discrete features. Specifically:

[0167] The video discrete features and video frame features are initially fused through the cross attention mechanism, and then concat operation is performed with the original features to obtain the fused video frame features f containing discrete features t :

[0168] f t =Concat(CrossAttention(D,x t ,x t ),x t )

[0169] Then the similarity between the original video frame and the discrete features is calculated and normalized to obtain the weight w of the video frame. t :

[0170] s t =sim(x t ,D),t∈[1,T]

[0171]

[0172] Where τ is a temperature parameter, s t represents the similarity between the t-th frame of the video and the discrete text, w t Represents the correlation between the t-th video frame and the discrete text after similarity normalization;

[0173] Adopt the confidence fusion strategy and use w t The value of is used as the confidence score for the fused video frame feature f t Weighted, and finally generate the final features of the video by summing

[0174]

[0175] It should be pointed out here that the device description in the above embodiment corresponds to the method description in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0176] The execution subjects of the above-mentioned processor and memory can be devices with computing functions such as computers, single-chip microcomputers, and microcontrollers. In specific implementation, the embodiments of the present invention do not limit the execution subjects and are selected according to the needs of actual applications.

[0177] The data signal is transmitted between the memory and the processor via a bus, which is not described in detail in the embodiment of the present invention.

[0178] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, the storage medium includes a stored program, and when the program is running, the device where the storage medium is located is controlled to execute the method steps in the above embodiment.

[0179] The computer-readable storage medium includes but is not limited to a flash memory, a hard disk, a solid-state drive, and the like.

[0180] It should be pointed out here that the description of the readable storage medium in the above embodiment corresponds to the description of the method in the embodiment, and the embodiment of the present invention will not be described in detail here.

[0181] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions according to the embodiments of the present invention are generated.

[0182] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions may be stored in a computer-readable storage medium or transmitted via a computer-readable storage medium. The computer-readable storage medium may be any available medium that can be accessed by the computer or a data storage device such as a server or a data center that includes one or more available media. The available medium may be a magnetic medium or a semiconductor medium, etc.

[0183] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.

[0184] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A video recognition method based on text-guided video to text discretization, characterized in that: The method comprises: Use CLIP's text and visual encoders to extract feature information from text and video frames respectively; Add a learnable prompt in the feature extraction process, fine-tune the features to generate learnable text features, and use the learnable text features to construct a dynamic dictionary; Get the dynamic dictionary and video features, and use the video-to-text discrete module to get the discrete features corresponding to the video; The discrete features and video frame features are input into the confidence fusion module together, and the discrete features and video frame features are initially fused to obtain the video frame features that aggregate the discrete features. The video frame features are further fused through the confidence score to obtain the feature expression of the entire video.

2. The video recognition method based on text-guided video to text discretization according to claim 1 is characterized in that: The method uses the similarity between frames and texts to obtain the text with the maximum similarity corresponding to each frame, and the texts corresponding to all video frames constitute candidate texts; then frame-level voting is performed, and the similarity is used as the voting weight. The text feature with the highest vote score is statistically calculated as the discrete feature corresponding to the video.

3. The video recognition method based on text-guided video to text discretization according to claim 1 is characterized in that: The process of adding a learnable prompt in the feature extraction process, fine-tuning the features to generate learnable text features, and constructing a dynamic dictionary using the learnable text features is as follows: Text encoder φ using CLIP t (·;θ t ) to extract text features; use the encoder parameters pre-trained by CLIP for initialization. For category text, add a learnable prompt before entering the text encoder; Add a randomly initialized learnable prompt before the text input to the text encoder to generate text features. The input y of the text encoder k for: y k =[w1][w2]…[w n ][class k ],k∈[1,K] Among them, K is the number of categories, w i is a learnable vector, and the text feature is c i for: c i =φ t (y k ;θ t ) Among them, θ t is the text encoder parameter; Constructing a dynamic dictionary using learnable text features extracted by a text encoder 4. The video recognition method based on text-guided video to text discretization according to claim 1 is characterized in that: The discrete features corresponding to the video are obtained by the discrete module from video to text: Visual encoder φ using CLIP v (·;θ v ) to extract video frame features; use the encoder parameters pre-trained by CLIP to initialize, for an input video V = {I t }, the video is evenly divided into T segments, and a random frame is selected from each segment to obtain T frames. The sampled video frames are input into the visual encoder in sequence. When inputting, each video frame is divided into A patch of size p×p is added before each frame with a randomly initialized learnable prompt, which is then input into the visual encoder to obtain the video frame features. The input s of the visual encoder is t for: s t =[u1][u2]…[u m ][e t ],t∈[1,T] Among them, T is the number of frames of each video, u i is a learnable vector, and the video feature x is obtained t for: x t =φ v (s t ;θ v ) Among them, θ v is the visual encoder parameter; Use a dynamic dictionary to measure the features of video frames. The measurement method is: in, It can be used as a discrete feature.

5. The video recognition method based on text-guided video to text discretization according to claim 2 is characterized in that: The candidate text is: [prd,ind]=argmax(S,0) Where prd represents the maximum confidence score of each frame, ind represents the corresponding index in the dictionary to which each frame should be assigned, and the index is used to obtain the candidate text features. The mask matrix M = one-hot (S :,ind[:] ) The rough classification results of all frames are counted, and all the obtained texts constitute the candidate texts.

6. The video recognition method based on text-guided video to text discretization according to claim 1, characterized in that: The discrete features and the video frame features are input into the confidence fusion module together, and the discrete features and the video frame features are preliminarily fused to obtain the video frame features that aggregate the discrete features. Specifically, the discrete features are: The video discrete features and video frame features are initially fused through the cross attention mechanism, and then concat operation is performed with the original features to obtain the fused video frame features f containing discrete features t : f t =Concat(CrossAttention(D,x t ,x t ),x t ) Then the similarity between the original video frame and the discrete features is calculated and normalized to obtain the weight w of the video frame. t : s t =sim(x t ,D),t∈[1,T] Where τ is a temperature parameter, s t represents the similarity between the t-th frame of the video and the discrete text, w t Represents the correlation between the t-th video frame and the discrete text after similarity normalization; Adopt the confidence fusion strategy and use w t The value of is used as the confidence score for the fused video frame feature f t Weighted, and finally generate the final features of the video by summing 7. A video recognition device based on text-guided video to text discretization, characterized in that: The device comprises: a processor and a memory, wherein program instructions are stored in the memory, and the processor calls the program instructions stored in the memory to enable the device to execute any one of the methods of claims 1-6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is enabled to perform the method according to any one of claims 1 to 6.

Images