A text-guided video-to-text discretization video recognition method and apparatus

Abstract

This invention discloses a video recognition method and apparatus based on text-guided video-to-text discrete processing. The method includes: using CLIP text and visual encoders to extract feature information from text and video frames respectively; adding a learnable prompt during feature extraction to fine-tune the features and generate learnable text features; constructing a dynamic dictionary using the learnable text features; acquiring the dynamic dictionary and video features; obtaining discrete features corresponding to the video through a video-to-text discrete processing module; inputting the discrete features and video frame features together into a confidence fusion module; performing preliminary fusion of the discrete features and video frame features to obtain video frame features aggregating the discrete features; and further fusion of the video frame features using confidence scores to obtain the feature representation of the entire video. The apparatus includes a processor and a memory. This invention achieves the fusion of video frame features and discrete text features, thereby obtaining a more accurate video feature description.

Description

Technical Field

[0001] This invention relates to the fields of multimodal image processing and video recognition, and in particular to a video recognition method and apparatus based on text-guided video-to-text discrete processing. Background Technology

[0002] Large-scale visual language models, pre-trained on extensive and meticulously curated image-text pair datasets, have made significant progress in artificial intelligence, successfully bridging the gap between vision and language. This has enhanced machines' ability to understand visual data and generate descriptions similar to human language content. Visual language models like CLIP have wide applications in various tasks, including image captioning, visual question answering, image-text retrieval, and multimodal generation. Given the success of these models in previous tasks, researchers hope to extend these pre-trained visual language models to video recognition. However, fully fine-tuning the model can lead to catastrophic forgetting. Therefore, researchers are exploring parameter-efficient fine-tuning methods as an ideal alternative to adapting pre-trained CLIP models for video recognition.

[0003] Videos composed of highly correlated frames can be effectively analyzed using pre-trained CLIP models. In recent years, numerous methods for video recognition have emerged. These methods can be broadly categorized into two types: prompt-based methods and adapter-based methods. Prompt-based methods use learnable visual and textual prompts to achieve video-text alignment. Prompt-based methods perform well on benchmark datasets, making prompt learning a widely used technique for efficient parameter fine-tuning. However, due to the limited number of parameters, they are only suitable for simple downstream tasks and cannot handle more complex challenges such as video recognition. Adapter-based methods, on the other hand, freeze the CLIP backbone network and add trainable layers on top of frame features for temporal modeling.

[0004] Previous CLIP-based methods were used to capture temporal dependencies across frames. However, it remains unclear whether frame features from CLIP contain temporal or similarity relationships, and the temporal modeling capabilities of temporal modules have not been validated. Furthermore, these methods use independent encoders for video-text contrastive learning, neglecting text representation in video modeling, thus limiting their generalization ability. How to better utilize text features and how to better construct temporal modeling modules are research topics that have emerged in recent years. Summary of the Invention

[0005] This invention provides a video recognition method and apparatus based on text-guided video-to-text discretization. The invention designs a feature extraction module based on the CLIP large model, using a learnable prompt to extract video and text features, and constructing a dynamic dictionary using text features. Through similarity calculation and a dynamic voting mechanism between video frames and text, the video is discretized onto text. A confidence feature fusion strategy is used to fuse video frame features with discrete text features, thereby obtaining a more accurate video feature description. See the description below for details:

[0006] A first aspect is a video recognition method based on text-guided video-to-text discrete text conversion, the method comprising:

[0007] Using CLIP's text and visual encoders, feature information is extracted from text and video frames, respectively;

[0008] A learnable prompt is added during the feature extraction process to fine-tune the features and generate learnable text features. The learnable text features are then used to construct a dynamic dictionary.

[0009] Obtain dynamic dictionary and video features, and obtain discrete features corresponding to the video through the video-to-text discrete module;

[0010] Discrete features and video frame features are input together into the confidence fusion module. The discrete features and video frame features are initially fused to obtain video frame features that aggregate the discrete features. The video frame features are then further fused using confidence scores to obtain the feature representation of the entire video.

[0011] The method utilizes the similarity between frames and text to obtain the text with the highest similarity for each frame, and the texts corresponding to all video frames constitute candidate texts; then, frame-level voting is performed, with similarity used as the voting weight, and the text feature with the highest vote score is the discrete feature corresponding to the video.

[0012] Specifically, the process of adding a learnable prompt during feature extraction, fine-tuning the features to generate learnable text features, and using these learnable text features to construct a dynamic dictionary involves:

[0013] Text encoder φ using CLIP t (·;θ t Text feature extraction is performed; the encoder parameters are initialized using CLIP pre-trained parameters, and for categorical text, a learnable prompt is added before the input text encoder;

[0014] A randomly initialized, learnable prompt is added before the text input to the text encoder to generate text features. The input y to the text encoder...k for:

[0015] y k =[w1][w2]…[w n ][class k ], k∈[1,K]

[0016] Where K is the number of categories, w i It is a learnable vector, and the text feature is c. i for:

[0017] c i =φ t (y k ;θ t )

[0018] Where, θ t For text encoder parameters;

[0019] A dynamic dictionary is constructed using learnable text features extracted by a text encoder.

[0020] The discrete features corresponding to the video obtained through the video-to-text discrete module are as follows:

[0021] Using CLIP's visual encoder φ v (·;θ v Perform video frame feature extraction; initialize using CLIP pre-trained encoder parameters, for an input video segment V = {I} t The method involves first dividing the video into T segments, then randomly selecting one frame from each segment to obtain T frames. These sampled video frames are then sequentially input into the visual encoder. During input, each video frame is segmented into... A patch of size p×p is created, and a randomly initialized, learnable prompt is added before each frame. This prompt is then input into a visual encoder to obtain video frame features. The input s of the visual encoder is... t for:

[0022] s t =[u1][u2]…[u m ][e t ],t∈[1,T]

[0023] Where T is the number of frames in each video, u i It is a learnable vector, from which video features x are obtained. t for:

[0024] x t =φ v (s t ;θ v )

[0025] Where, θ v For visual encoder parameters;

[0026] The features of video frames are measured using a dynamic dictionary. The measurement method is as follows:

[0027]

[0028] in, It can be used as a discrete feature.

[0029] The candidate text is:

[0030] [prd,ind] = argmax(S,0)

[0031] Where prd represents the maximum confidence score of each frame, and ind represents the corresponding index in the dictionary to which each frame should be assigned. This index is used to obtain candidate text features, using a mask matrix M = one-hot(S :,ind[:] The coarse classification results of all frames are statistically analyzed, and all the texts obtained constitute candidate texts.

[0032] Specifically, the step of inputting discrete features and video frame features together into the confidence fusion module, and performing preliminary fusion of discrete features and video frame features to obtain video frame features that aggregate discrete features, involves:

[0033] The discrete video features and video frame features are initially fused using a cross-attention mechanism, and then concatted with the original features to obtain fused video frame features f that include the discrete features. t :

[0034] f t =Concat(CrossAttention(D,x) t ,x t ),x t )

[0035] Then, similarity is calculated between the original video frames and discrete features, and the weights w of the video frames are obtained by normalization. t :

[0036] s t =sim(x t ,D),t∈[1,T]

[0037]

[0038] Where τ is a temperature parameter, s t w represents the similarity between the t-th video frame and the discrete text. t This represents the correlation between the t-th video frame and the discrete text after similarity normalization.

[0039] A confidence fusion strategy is adopted, and w is used. t The value of f is used as a confidence score for the fused video frame features. t The weighted summation is then used to generate the final features of the video.

[0040]

[0041] A second aspect is a text-guided video-to-text discrete video recognition device, the device comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the method described in any one of the first aspects.

[0042] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of the first aspects.

[0043] The beneficial effects of the technical solution provided by this invention are:

[0044] 1. This invention proposes a text-guided video-to-text discretization method, which can discretize video features onto text, thereby leveraging the stability and accuracy of text features to improve the model's representation of video features and enhance its generalization ability.

[0045] 2. This invention designs a dynamic dictionary based on a text encoder, which uses the learnable text features obtained by the encoder to construct the dynamic dictionary, reducing the difficulty of building a dynamic dictionary from scratch, reducing the amount of training, and avoiding potential conflicts or collapse problems in the future.

[0046] 3. This invention proposes a feature fusion method based on confidence scores, which initially fuses discrete text features, integrates the advantages of text features into video features, and assigns different confidence scores to video frames based on similarity calculation, which can effectively reduce the interference of video redundancy or noise information on video recognition. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the overall network structure;

[0048] Figure 2 This is a flowchart of a text-guided video-to-text discretization video recognition method. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0050] To address the problems existing in the background technology, this invention proposes a text-guided video-to-text discrete video recognition method. This method integrates the stability of text features and their clear semantic information into video features, thereby improving the model's expression of video features and enhancing its video recognition capabilities. The network includes a feature extraction module, a video-to-text discrete module, and a confidence fusion module. This video recognition method avoids the uncertainties of temporal modeling and can better utilize text features to improve video recognition performance.

[0051] Example 1

[0052] This invention provides a text-guided video-to-text discretization video recognition method, which includes the following steps:

[0053] 101: The feature extraction module uses CLIP's text and visual encoders, which leverage powerful multimodal learning capabilities to extract rich feature information from text and video frames, respectively; in addition, this module is initialized using CLIP's pre-trained parameters.

[0054] The above steps enable the feature extraction process to not only have good visual and linguistic feature alignment capabilities, but also to fully utilize the generalization capabilities learned by the pre-trained model on large-scale data.

[0055] 102: Learnable prompts are added during feature extraction to fine-tune the features. Learnable prompts are added to the first 9 layers of the encoder to generate learnable text features, making the text and visual features more aligned. The learnable text features are then used to construct a dynamic dictionary.

[0056] 103: Obtain dynamic dictionary and video features. Through the video-to-text discrete module, obtain the discrete features corresponding to the video.

[0057] After obtaining the dictionary and video frame features, the text with the highest similarity between the frame and the text is obtained by using the similarity between the frame and the text. The text corresponding to all video frames constitutes the candidate text. After obtaining the candidate text, frame-level voting is carried out, with similarity as the voting weight. The text feature with the highest vote score is the discrete feature corresponding to the video.

[0058] 104: Obtain discrete features and input them together with video frame features into the confidence fusion module. Perform preliminary fusion of discrete features and video frame features to obtain video frame features that aggregate discrete features. Then, further fuse the video frame features through confidence scores to obtain the feature representation of the entire video.

[0059] In summary, this method, through a video recognition model composed of three modules—feature extraction, video-to-text discretization, and confidence fusion—can integrate the stability of text features and their clear semantic information into video features, thereby enhancing the model's ability to express video features. This video recognition method avoids the uncertainties of temporal modeling and can better utilize text features to improve video recognition performance.

[0060] Example 2

[0061] The scheme in Example 1 will be further described below with specific examples and calculation formulas:

[0062] I. Data Preparation

[0063] The effectiveness of the proposed method was verified on four datasets: HMDB-51, UCF-101, SSv2, and Kinetics-400.

[0064] The HMDB51 dataset is a widely used benchmark dataset for video action recognition tasks, containing video clips from real-world scenes. The dataset consists of 51 different action categories, covering a wide range of complex actions and scenarios, such as dancing, running, punching, and cycling. Each action category contains approximately 100 videos, providing a total of about 6,800 video clips.

[0065] The UCF101 dataset contains 101 different action categories, covering a wide range of everyday actions, including sports (e.g., basketball, soccer), daily activities (e.g., cleaning, drinking water), musical performances (e.g., playing the piano, drums), and stunt performances. The dataset contains 13,320 video clips, typically ranging from a few seconds to tens of seconds in length. The action categories in the UCF101 dataset exhibit similarities (e.g., various ball sports), while the video backgrounds also vary considerably. Furthermore, some videos in UCF101 are of low quality, adding further difficulty to video recognition.

[0066] The SSv2 (Something-Something V2) dataset is a benchmark dataset specifically designed for video action understanding and temporal modeling tasks. Unlike traditional video datasets, the SSv2 dataset focuses on capturing actions and their temporal relationships through short videos, making it suitable for studying the importance of temporal features for action recognition. The SSv2 dataset contains 174 different action categories, with over 220,000 video samples, each video ranging from 2 to 6 seconds in length, and a large number of videos within each category.

[0067] Kinetics-400 contains 400 different action categories, covering a wide range of daily activities, sports, recreational activities, and musical instrument playing. Examples include categories like "playing soccer," "cycling," "playing the piano," and "dancing." These categories not only involve human movements but also interactions between people and objects, and between people and their environment. The dataset contains over 240,000 video clips, all from YouTube, typically around 10 seconds long. The video clips are of high quality, with a sufficient number of video samples for each action category, usually 400-600 clips per category. The training set contains approximately 240,000 video clips, while the validation and test sets each contain approximately 20,000 video clips.

[0068] II. Text-Guided Video-to-Text Discrete-to-Video Recognition Model

[0069] The text-guided video-to-text discretization-to-video recognition model in this embodiment of the invention, such as... Figure 1 As shown, it includes three modules: a feature extraction module, a video-to-text discrete module, and a confidence fusion module.

[0070] In this embodiment of the invention, the feature extraction module employs CLIP's text and visual encoders. These encoders leverage powerful multimodal learning capabilities to extract rich feature information from text and video frames, respectively. Furthermore, the module is initialized using CLIP's pre-trained parameters, which enables the feature extraction process to not only possess excellent visual and linguistic feature alignment capabilities but also fully utilize the generalization capabilities learned by the pre-trained model on large-scale data. To better generate video frame and text features, this embodiment adds learnable prompts to the input parts of both encoders and constructs a dynamic dictionary using learnable text features. The acquired visual features and dynamic dictionary are fed into the video-to-text discrete module. Through similarity calculation, coarse classification information corresponding to the video frames is selected. The coarse classification information of all frames constitutes candidate texts. Voting is performed using the similarity scores between frames and candidate texts, and the candidate text feature with the highest total score becomes the discrete feature of the video. The discrete features and video frame features are initially fused using an attention mechanism and jump links. Then, the confidence level of the video frame fusion is obtained using the similarity between the video frames and discrete features. Based on the confidence level, the video frame features are fused to obtain the entire video feature. Finally, the obtained video features are used for video classification and recognition, and the entire model is trained using contrastive learning.

[0071] The steps for constructing a dynamic dictionary are as follows:

[0072] Text encoder φ using CLIP t (·;θ t Text feature extraction is performed; the encoder parameters are initialized using CLIP pre-trained parameters, and for categorical text, a learnable prompt is added before the input text encoder;

[0073] In the text feature extraction process, a randomly initialized, learnable prompt is first added before the text input to the text encoder to generate text features. The input y to the text encoder... k for:

[0074] y k =[w1][w2]…[w n ][class k ], k∈[1,K] (1)

[0075] Where K is the number of categories, w i It is a learnable vector, and the text feature is c. i for:

[0076] c i =φ t (y k ;θ t (2)

[0077] Where, θ t These are the parameters for the text encoder. By adding a learnable prompt, the model can capture variable contextual information, allowing it to better adapt to different task requirements.

[0078] In the process of constructing a dynamic dictionary, predefined category texts can effectively represent visual clusters and reduce conflicts. Therefore, embodiments of the present invention use learnable text features extracted by a text encoder to construct a dynamic dictionary. With this design, the model can not only achieve accurate feature mapping between different categories, but also automatically adjust the dictionary during training to gradually adapt to the changing features in the video data, thereby enhancing the matching ability between video and text.

[0079] The implementation steps of the text-guided video-to-text discretization mechanism are as follows: First, using CLIP's visual encoder φ v (·;θ v Perform video frame feature extraction; initialize using CLIP pre-trained encoder parameters, for an input video segment V = {I} t},in The method involves first dividing the video into T uniform segments, then randomly selecting one frame from each segment to obtain T frames. These sampled video frames are then sequentially input into the visual encoder. During input, each video frame is segmented into... A patch of size p×p is created, and a randomly initialized, learnable prompt is added before each frame. This prompt is then input into a visual encoder to obtain video frame features. The input s of the visual encoder is... t for:

[0080] s t =[u1][u2]…[u m ][e t ], t∈[1,T] (3)

[0081] Where T is the number of frames in each video, u i It is a learnable vector, from which video features x are obtained. t for:

[0082] x t =φ v (s t ;θ v ) (4)

[0083] Where, θ vThese are the parameters for the visual encoder. By adding these cue words before each frame, the image encoder can not only understand each video frame from different perspectives, but also obtain additional contextual information through the cue words, thereby enhancing the model's performance in video sequences.

[0084] Obtaining a dynamic dictionary and video frame features x t Then, similar to VQ-VAE, a dynamic dictionary can be used to measure the features of video frames. The measurement method is as follows:

[0085]

[0086] in, It can be used as a discrete feature.

[0087] Based on this, the embodiments of the present invention adjust the measurement method. First, the similarity s between the t-th video frame and the k-th text is calculated using cosine similarity. tk :

[0088] s tk =sim(x t ,c k ),t∈[1,T],k∈[1,K] (6)

[0089] Where sim(·) is the cosine similarity calculation, x t For video frame features, c k Let K be the text features, K be the number of text categories, and T be the number of video frames, which together form the similarity matrix S.

[0090] Then, coarse classification of video frames is performed. The text feature with the highest similarity between each video frame and the text is obtained as the coarse classification for that frame. The calculation method is as follows:

[0091] [prd,ind]=argmax(S,0) (7)

[0092] Where prd represents the maximum confidence score of each frame, and ind represents the corresponding index in the dictionary to which each frame should be assigned, which is used to obtain candidate text features. Then, the mask matrix M = one-hot(S :,ind[:] The coarse classification results of all frames are statistically analyzed, and all the texts obtained constitute candidate texts.

[0093] Then, the similarity value of each frame to its coarse classification is calculated as the voting result for that frame. The final score of all candidate texts is then calculated, and the text feature with the highest score is the discrete feature of the video. The voting result is calculated as follows:

[0094] [pm,i]=max(sum(M⊙S,-1)) (8)

[0095] Where i is the index of the discrete text feature corresponding to the video, i.e., D = c i It is a discrete feature.

[0096] The implementation steps of the confidence-based feature fusion strategy are as follows:

[0097] After obtaining the discrete video features, they are initially fused with the video frame features using a cross-attention mechanism, and then concatenated with the original features to obtain the fused video frame features f containing the discrete features. t ;

[0098] f t =Concat(CrossAttention(D,x) t ,x t ),x t (9)

[0099] Then, similarity is calculated between the original video frames and discrete features, and then normalized to obtain the weight w of the video frames. t :

[0100] s t =sim(x t ,D),t∈[1,T] (10)

[0101]

[0102] Where τ is a temperature parameter, s t w represents the similarity between the t-th video frame and the discrete text. t This represents the relevance between the t-th video frame and the discrete text after similarity normalization. This embodiment of the invention employs a confidence fusion strategy and uses w... t The value of f is used as a confidence score for the fused video frame features. t The weighted summation is then used to generate the final features of the video.

[0103]

[0104] In contrastive training, this embodiment of the invention maximizes the cosine similarity between the video representation and its corresponding text representation within each batch through cross-entropy loss, in order to fine-tune the CLIP model. The contrastive learning optimization objective is:

[0105]

[0106] III. Evaluation Indicators and Agreements

[0107] For all trackers, this embodiment of the invention uses success rate (Acc) and accuracy rate (Pre) for evaluation. For a certain category, this embodiment of the invention can denote correctly detected target data as TP (TruePositive), data that is incorrectly identified as a target as FP (FalsePositive), data that is incorrectly identified as a non-target as FN (FalseNegatives), and data that is correctly detected as a non-target as TN (TrueNegatives), thus obtaining the confusion matrix as shown in Table 1.

[0108] Table 1 Confusion Matrix

[0109]

[0110]

[0111] Acc represents the proportion of all correctly classified results out of the total observations.

[0112]

[0113] Pre is the proportion of correct model predictions among all outcomes where the model prediction is P.

[0114] IV. Details on Model Usage

[0115] 1. Data Augmentation: Due to limited computing resources, a strategy of randomly flipping and cropping training video frames was adopted to increase the diversity of training data. For video frames larger than 680×640, the size of the video frames was first adjusted to be smaller than 680×640. Furthermore, the accuracy of annotations was improved by manually adding annotations.

[0116] 2. Model optimization:

[0117] This method sets the batch size N to 64 during training, and then uses the Adam optimization algorithm, achieving a batch size of 8×10⁻⁶ for the first 50 epochs. -4 The learning rate is used to train the network.

[0118] 3. Video recognition:

[0119] The feature extraction module in this embodiment of the invention can obtain categorical text features and video frame features. The categorical text features include all text categories in the video, and the obtained text features constitute a dynamic dictionary. The video frame features are obtained through a video-to-text discretization module, which then fuses the discrete features with the video frame features. The resulting features are further fused using confidence scores to obtain the video feature representation. This feature representation includes both overall video features and discrete text information, improving the model's ability to express video features. The model training in this embodiment of the invention uses a contrastive learning method.

[0120] The embodiments of the present invention have the following three key inventive points:

[0121] I. A video-to-text discretization video recognition model is proposed. The model improves its ability to represent video features by discretizing the video onto text and integrating the stability and clear semantic information of text features into the video features. This video recognition model avoids the uncertainties of temporal modeling and can better utilize text features to improve video recognition performance.

[0122] II. Propose a dynamic dictionary construction method

[0123] Technical Effects: This embodiment of the invention employs a dynamic dictionary built based on a text encoder, utilizing learnable text features generated by the encoder to construct the dynamic dictionary. This method reduces the complexity of building a dictionary from scratch, decreases the workload required for training, and effectively avoids potential conflicts or collapse issues later on.

[0124] III. Proposing a feature fusion strategy based on confidence scores

[0125] Technical effect: By initially fusing discrete features and video frame features, the embodiments of the present invention can integrate the advantages of text features into video features. Furthermore, by using a confidence score fusion method based on similarity calculation, different confidence levels are assigned to video frames, which can effectively reduce the interference of video redundancy or noise information on video recognition.

[0126] In summary, this invention proposes a video-to-text discrete video recognition model. By transforming video features into text features, it fully leverages the stability and clear semantic information of text features, thereby enhancing the model's ability to express video features and avoiding the uncertainties of temporal modeling. Simultaneously, a dynamic dictionary method based on a text encoder is proposed. This method utilizes learnable text features to construct a dynamic dictionary, simplifying the dictionary construction process, reducing training complexity, and effectively avoiding potential conflicts and collapse issues. Furthermore, this invention designs a feature fusion strategy based on confidence scores. By fusing discrete features and video frame features, and using similarity calculations to assign different confidence scores to video frames, it effectively reduces the impact of redundant and noisy information on video recognition performance.

[0127] Example 3

[0128] This invention proposes comparisons with multiple video recognition methods on the HMDB-51, UCF-101, SSv2, and Kinetics-400 datasets. The comparisons are made with video recognition methods based on large-scale image pre-trained models, such as Uniformer, Swin, ViViT, TokenLearner, and MViTv2; and with video recognition methods based on fine-tuning of large pre-trained models, such as ActionCLIP, XCLIP, ViFi-CLIP, ST-Adapter, STAN-conv, Vita-CLIP, and M2-CLIP.

[0129] The experimental results on the Kinetics-400 dataset are shown in Table 1. The Top-1, Top-5, Avg, and GFLOPs scores are 85.1, 97.1, 91.1, and 194, respectively, demonstrating significant competitiveness compared to other comparative methods. In the Top-1, Top-5, and Avg metrics, the method of this invention outperforms other comparative methods. Therefore, it can be proven that the method in this invention can better analyze video content, better utilize the advantages of text features, and improve the model's ability to express and recognize video features.

[0130] The results of few-shot experiments on the HMDB-51, UCF-101, and SSv2 datasets are shown in Table 2. On the HMDB-51 dataset, the Top-1 accuracy for k=2, k=4, k=8, and k=16 is 57.9, 62.8, 65.6, and 68.0, respectively. On the UCF-101 dataset, the Top-1 accuracy for k=2, k=4, k=8, and k=16 is 82.4, 86.9, 91.3, and 93.0, respectively. On the SSv2 dataset, the Top-1 accuracy for k=2, k=4, k=8, and k=16 is 7.1, 8.7, 10.4, and 13.0, respectively. It can be seen that the recognition accuracy of the proposed method gradually increases with the number of videos, and is consistently higher than the comparative methods. This demonstrates that the method in this invention can achieve good results even with a small number of training samples, further validating the effectiveness of the proposed method.

[0131] The experimental results used in this embodiment are shown in Table 3. These results demonstrate the performance of three variants of the proposed method on the UCF-101 dataset: VTD-CLIP(no), VTD-CLIP(fix), and VTD-CLIP(w / oconf). VTD-CLIP(no) indicates that no learnable prompt was added to the text input; VTD-CLIP(fix) indicates that a fixed prompt, "a photo of a," was used; and VTD-CLIP(w / o conf) does not use confidence fusion. All variants were trained on the training set and tested on the test set. The training steps and other parameters were identical across the different experiments, and the evaluation was the same. As shown in Table 3, the proposed method achieved better results than its variants, and simultaneously verified that dynamic dictionaries and confidence fusion can significantly improve the model's representation of video features, thereby enhancing the model's performance in video recognition tasks.

[0132] Table 1

[0133]

[0134] Table 2

[0135]

[0136] Table 3

[0137]

[0138]

[0139] Example 4

[0140] A text-guided video-to-text discrete video recognition device includes a processor and a memory. The memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the device to perform the following method steps in Embodiment 1:

[0141] Using CLIP's text and visual encoders, feature information is extracted from text and video frames, respectively;

[0142] A learnable prompt is added during the feature extraction process to fine-tune the features and generate learnable text features. The learnable text features are then used to construct a dynamic dictionary.

[0143] Obtain dynamic dictionary and video features, and obtain discrete features corresponding to the video through the video-to-text discrete module;

[0144] Discrete features and video frame features are input together into the confidence fusion module. The discrete features and video frame features are initially fused to obtain video frame features that aggregate the discrete features. The video frame features are then further fused using confidence scores to obtain the feature representation of the entire video.

[0145] The process involves using the similarity between frames and text to obtain the text with the highest similarity for each frame, and then using the text corresponding to all video frames to form candidate texts. Frame-level voting is then conducted, with similarity used as the voting weight, and the text feature with the highest vote score is the discrete feature corresponding to the video.

[0146] Specifically, the process of adding a learnable prompt during feature extraction, fine-tuning the features to generate learnable text features, and using these learnable text features to construct a dynamic dictionary involves the following steps:

[0147] Text encoder φ using CLIP t (·;θ t Text feature extraction is performed; the encoder parameters are initialized using CLIP pre-trained parameters, and for categorical text, a learnable prompt is added before the input text encoder;

[0148] A randomly initialized, learnable prompt is added before the text input to the text encoder to generate text features. The input y to the text encoder... k for:

[0149] y k =[w1][w2]…[w n ][class k ], k∈[1,K]

[0150] Where K is the number of categories, w i It is a learnable vector, and the text feature is c.i for:

[0151] c i =φ t (y k ;θ t )

[0152] Where, θ t For text encoder parameters;

[0153] A dynamic dictionary is constructed using learnable text features extracted by a text encoder.

[0154] Specifically, the discrete features corresponding to the video are obtained through the video-to-text discrete module:

[0155] Using CLIP's visual encoder φ v (·;θ v Perform video frame feature extraction; initialize using CLIP pre-trained encoder parameters, for an input video segment V = {I} t The method involves first dividing the video into T segments, then randomly selecting one frame from each segment to obtain T frames. These sampled video frames are then sequentially input into the visual encoder. During input, each video frame is segmented into... A patch of size p×p is created, and a randomly initialized, learnable prompt is added before each frame. This prompt is then input into a visual encoder to obtain video frame features. The input s of the visual encoder is... t for:

[0156] s t =[u1][u2]…[u m ][e t ],t∈[1,T]

[0157] Where T is the number of frames in each video, u i It is a learnable vector, from which video features x are obtained. t for:

[0158] x t =φ v (s t ;θ v )

[0159] Where, θ v For visual encoder parameters;

[0160] The features of video frames are measured using a dynamic dictionary. The measurement method is as follows:

[0161]

[0162] in, It can be used as a discrete feature.

[0163] The candidate texts are:

[0164] [prd,ind] = argmax(S,0)

[0165] Where prd represents the maximum confidence score of each frame, and ind represents the corresponding index in the dictionary to which each frame should be assigned. This index is used to obtain candidate text features, using a mask matrix M = one-hot(S :,ind[:] The coarse classification results of all frames are statistically analyzed, and all the texts obtained constitute candidate texts.

[0166] Specifically, discrete features and video frame features are input together into the confidence fusion module, and the discrete features and video frame features are initially fused to obtain video frame features that aggregate the discrete features.

[0167] The discrete video features and video frame features are initially fused using a cross-attention mechanism, and then concatted with the original features to obtain fused video frame features f that include the discrete features. t :

[0168] f t =Concat(CrossAttention(D,x) t ,x t ),x t )

[0169] Then, similarity is calculated between the original video frames and discrete features, and the weights w of the video frames are obtained by normalization. t :

[0170] s t =sim(x t ,D),t∈[1,T]

[0171]

[0172] Where τ is a temperature parameter, s t w represents the similarity between the t-th video frame and the discrete text. t This represents the correlation between the t-th video frame and the discrete text after similarity normalization.

[0173] A confidence fusion strategy is adopted, and w is used. t The value of f is used as a confidence score for the fused video frame features. t The weighted summation is then used to generate the final features of the video.

[0174]

[0175] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.

[0176] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.

[0177] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.

[0178] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium, the storage medium including a stored program, which, when the program is running, controls the device where the storage medium is located to execute the method steps in the above embodiments.

[0179] The computer-readable storage medium includes, but is not limited to, flash memory, hard disk, solid-state drive, etc.

[0180] It should be noted that the description of the readable storage medium in the above embodiments corresponds to the description of the method in the embodiments, and the embodiments of the present invention will not be repeated here.

[0181] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated.

[0182] A computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in or transmitted through a computer-readable storage medium. A computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic or semiconductor, etc.

[0183] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.

[0184] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video recognition method based on text-guided video-to-text discretization, characterized in that, The method includes: Using CLIP's text and visual encoders, feature information is extracted from text and video frames, respectively; A learnable prompt is added during the feature extraction process to fine-tune the features and generate learnable text features. The learnable text features are then used to construct a dynamic dictionary. Obtain dynamic dictionary and video features, and obtain discrete features corresponding to the video through the video-to-text discrete module; Discrete features and video frame features are input together into the confidence fusion module. The discrete features and video frame features are initially fused to obtain video frame features that aggregate the discrete features. The video frame features are then further fused using confidence scores to obtain the feature representation of the entire video.

2. The video recognition method based on text-guided video-to-text discretization according to claim 1, characterized in that, The method utilizes the similarity between frames and text to obtain the text with the highest similarity for each frame, and the texts corresponding to all video frames constitute candidate texts; then, frame-level voting is performed, with similarity used as the voting weight, and the text feature with the highest vote score is the discrete feature corresponding to the video.

3. The video recognition method based on text-guided video-to-text discrete processing according to claim 1, characterized in that, The specific steps of adding a learnable prompt during feature extraction, fine-tuning the features to generate learnable text features, and using these learnable text features to construct a dynamic dictionary are as follows: Text encoder φ using CLIP t (·;θ t Text feature extraction is performed; the encoder parameters are initialized using CLIP pre-trained parameters, and for categorical text, a learnable prompt is added before the input text encoder; A randomly initialized, learnable prompt is added before the text input to the text encoder to generate text features. The input y to the text encoder... k for: y k =[w1][w2]…[w n ][class k ],k∈[1,K] Where K is the number of categories, w i It is a learnable vector, and the text feature is c. i for: c i =φ t (y k ;θ t ) Where, θ t For text encoder parameters; A dynamic dictionary is constructed using learnable text features extracted by a text encoder.

4. The video recognition method based on text-guided video-to-text discretization according to claim 1, characterized in that, The discrete features corresponding to the video are obtained through the video-to-text discrete module as follows: Using CLIP's visual encoder φ v (·;θ v Perform video frame feature extraction; initialize using CLIP pre-trained encoder parameters, for an input video segment V = {I} t The method involves first dividing the video into T segments, then randomly selecting one frame from each segment to obtain T frames. These sampled video frames are then sequentially input into the visual encoder. During input, each video frame is segmented into... A patch of size p×p is created, and a randomly initialized, learnable prompt is added before each frame. This prompt is then input into a visual encoder to obtain video frame features. The input s of the visual encoder is... t for: s t =[u1][u2]…[u m ][e t ],t∈[1,T] Where T is the number of frames in each video, u i It is a learnable vector, from which video features x are obtained. t for: x t =φ v (s t ;θ v ) Where, θ v For visual encoder parameters; The features of video frames are measured using a dynamic dictionary. The measurement method is as follows: in, It can be used as a discrete feature.

5. The video recognition method based on text-guided video-to-text discrete processing according to claim 2, characterized in that, The candidate text is: [prd,ind] = argmax(S,0) Where prd represents the maximum confidence score of each frame, and ind represents the corresponding index in the dictionary to which each frame should be assigned. This index is used to obtain candidate text features, using a mask matrix M = one-hot(S :,ind[:] The coarse classification results of all frames are statistically analyzed, and all the texts obtained constitute candidate texts.

6. The video recognition method based on text-guided video-to-text discretization according to claim 1, characterized in that, The step of inputting discrete features and video frame features together into the confidence fusion module, and initially fusing the discrete features and video frame features to obtain video frame features aggregating discrete features, specifically involves: The discrete video features and video frame features are initially fused using a cross-attention mechanism, and then concatted with the original features to obtain fused video frame features f that include the discrete features. t : f t =Concat(CrossAttention(D,x t ,x t ),x t ) Then, similarity is calculated between the original video frames and discrete features, and the weights w of the video frames are obtained by normalization. t : s t =sim(x t ,D),t∈[1,T] Where τ is a temperature parameter, s t w represents the similarity between the t-th video frame and the discrete text. t This represents the correlation between the t-th video frame and the discrete text after similarity normalization. A confidence fusion strategy is adopted, and w is used. t The value is used as a confidence score for the fused video frame features f. t The weighted summation is then used to generate the final features of the video.

7. A video recognition device based on text-guided video-to-text discrete processing, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor invoking the program instructions stored in the memory to cause the device to perform the method according to any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method described in any one of claims 1-6.

Images