A CLIP-based video text embedding fusion classification method

Abstract

This invention relates to the field of video action recognition, specifically a CLIP-based video text embedding fusion classification method. The method extracts text features and video frame-level visual features using a pre-trained CLIP model's text encoder and visual encoder, respectively. It then performs multi-scale adaptive feature fusion and parameter-free extremum feature fusion operations on the video frame-level visual features, and applies multi-level cross-modal attention fusion operations to the generated video-level embedding vectors to ultimately generate a global video-level embedding vector. Finally, it calculates the maximum cosine similarity between the global video-level embedding vector and the text embedding vector. This invention reduces the computational cost of training and inference while preserving key information in automatically focusing on video frames, and has broad application prospects.

Description

Technical Field

[0001] This invention relates to a video text embedding fusion classification method based on CLIP, belonging to the field of video action recognition. Background Technology

[0002] With the rapid development of information technology, the generation of video content is growing at an unprecedented rate. Video classification, as a fundamental task of video content understanding, is of great significance for various fields such as video retrieval, content recommendation, and intelligent surveillance. However, the diversity and complexity of video content pose significant challenges to automatic classification. Traditional video classification methods typically rely solely on the visual information of the video, ignoring the rich textual descriptions that may exist within it. This approach fails to fully utilize the semantic meaning of the textual information, resulting in limited classification accuracy. With the rise of deep learning, multimodal learning has also become an important direction in video classification, enhancing classification performance by combining visual information and relevant textual information. However, it still suffers from shortcomings such as the inability to effectively fuse video frame-level features and the lack of cross-modal attention fusion.

[0003] The CLIP model is a multimodal pre-trained neural network. Its core idea is to pre-train using a large amount of paired image and text data to learn the alignment relationships between images and text. The CLIP model consists of two main parts: a Text Encoder and an Image Encoder. The Text Encoder converts text into a low-dimensional vector representation, while the Image Encoder converts images into similar vector representations. During the prediction phase, the CLIP model generates predictions by calculating the cosine similarity between the text and image vectors. This model is particularly suitable for zero-shot learning tasks, meaning the model can make predictions without seeing new training examples of images or text. The CLIP model's Image Encoder uses ResNet50 as its foundation and improves upon it by using attention pooling instead of global average pooling layers.

[0004] CLIP models include functions such as image-text retrieval and image-text generation, and they perform well in multiple fields. Examples include Chinese patent application CN117112838A, "A Video-Text Retrieval Method Based on CLIP Contrast Learning," and Chinese patent application CN116628262A, "A Video-Text Retrieval Method Based on Knowledge Enhancement."

[0005] The CLIP model also supports classification tasks, enabling it to classify images by providing textual descriptions of image categories. However, the CLIP model also has some limitations, such as underperforming on certain specific models on fine-grained classification datasets, inefficient data utilization, and the potential to learn harmful information. Summary of the Invention

[0006] To address the limitations of existing technologies, this invention proposes a CLIP-based video-text embedding fusion classification method. This method aims to fully utilize video and text information, employing multi-scale adaptive feature fusion and multi-level cross-modal attention fusion to more accurately understand the semantic meaning between video and text, capture dynamic changes and contextual information in the video, thereby improving the model's classification performance and generalization ability.

[0007] This method uses a classification model to process videos and output predicted values; the classification model is a CLIP-based video-text embedding fusion classification model; the construction and processing steps of the classification model include:

[0008] Step S1: Build a text encoder and a visual encoder based on the pre-trained model CLIP to independently extract text features of labels and visual features at the video frame level.

[0009] Step S2, use a text encoder to independently encode the category label Y and wrap it in the prompt template "aphoto of <y>In the process, the final text embedding vector t is generated;

[0010] Step S3, use a visual encoder to process video samples V with T frames. i The images are independently encoded into a batch, and a batch of video frame-level embedding vector representations X are generated. i ;

[0011] Step S4: The batch of video frame-level embedding vectors X generated by the visual encoder... i Multi-scale adaptive feature fusion operation is performed to generate video-level embedding vector A1, parameterless maximum value feature fusion operation is performed to generate video-level embedding vector A2, and parameterless minimum value feature fusion operation is performed to generate video-level embedding vector A3.

[0012] Step S5: Perform multi-level cross-modal attention fusion operation on the video-level embedding vectors A1, A2, and A3 generated in step S4 to finally generate the global video-level embedding vector V. global ;

[0013] Step S6: Calculate the global video-level embedding vector V generated in step S5. global The maximum cosine similarity between the text embedding vector t generated in step S2 and CLIP's text encoder and visual encoder is fine-tuned using the cross-entropy loss function.

[0014] Video frame level refers to extracting the visual feature information of each individual frame, without including the temporal context information across frames.

[0015] Video-level features extraction focuses on the visual features of the entire video sequence, rather than just a single image frame.

[0016] The beneficial effects of this invention are:

[0017] (1) This invention proposes multi-scale adaptive feature fusion, parameter-free maximum feature fusion, and parameter-free minimum feature fusion, which fully consider the correlation and temporal information between video frame-level features. This feature fusion method can better capture dynamic changes and contextual information in the video.

[0018] (2) By proposing a multi-level cross-modal attention fusion, this invention can effectively integrate information from different methods to generate global video-level embedding vectors, thereby improving the accuracy and richness of feature representation.

[0019] This invention relates to the field of video action recognition, specifically a video-text embedding fusion classification method based on CLIP. This method extracts text features and video frame-level visual features using the pre-trained CLIP model's text encoder and visual encoder, respectively. It then performs multi-scale adaptive feature fusion and parameter-free extremum feature fusion operations on the video frame-level visual features, and applies multi-level cross-modal attention fusion operations to the generated video-level embedding vectors to finally generate a global video-level embedding vector. Finally, it calculates the maximum cosine similarity between the global video-level embedding vector and the text embedding vector. This invention reduces the computational cost of training and inference while ensuring automatic focusing on key information in video frames, and has broad application prospects. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0021] Figure 1 This is a flowchart illustrating a CLIP-based video text embedding fusion classification method in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the structure of a video text embedding fusion classification method model based on CLIP in an embodiment of the present invention.

[0023] Figure 3 This is a schematic diagram of the multi-scale adaptive feature fusion structure in an embodiment of the present invention.

[0024] Figure 4 This is an experimental result diagram of a video text embedding fusion classification method model based on CLIP in an embodiment of the present invention. Detailed Implementation

[0025] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0026] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0027] refer to Figure 1 The method in this example includes the following steps:

[0028] Step S1: Build a text encoder and a visual encoder based on the pre-trained model CLIP to independently extract text features of labels and visual features at the video frame level.

[0029] Step S2, use a text encoder to independently encode the category label Y and wrap it in the prompt template "aphoto of <y>In the process, the final text embedding vector t is generated;

[0030] Step S3, use a visual encoder to process video samples V with T frames. i The images are independently encoded into a batch, and a batch of video frame-level embedding vector representations X are generated. i ;

[0031] Step S4: The batch of video frame-level embedding vectors X generated by the visual encoder... i Multi-scale adaptive feature fusion operation is performed to generate video-level embedding vector A1, parameterless maximum value feature fusion operation is performed to generate video-level embedding vector A2, and parameterless minimum value feature fusion operation is performed to generate video-level embedding vector A3.

[0032] Step S5: Perform multi-level cross-modal attention fusion operation on the generated video-level embedding vectors A1, A2, and A3 to finally generate the global video-level embedding vector V. global ;

[0033] Step S6, process the generated global video-level embedding vector V... global The maximum cosine similarity between CLIP and the text embedding vector t is calculated, and the text encoder and visual encoder of CLIP are fine-tuned by the cross-entropy loss function.

[0034] In this embodiment, the multi-scale adaptive feature fusion operation in step S4, such as Figure 3 As shown, its calculation method includes:

[0035]

[0036] A1=Avg((δ u (X i W u )⊙A tt δ v (X i W v ))W o )

[0037] Among them, X i W represents a batch of video frame-level embedding vectors generated by the visual encoder. z W u and W v They represent X respectively i The first learnable weight matrix, the second learnable weight matrix, and the third learnable weight matrix, δ z δ u and δ v Let Q represent the first activation function, the second activation function, and the third activation function, respectively. The transformation is a translation operation on the components of the feature vector, where R represents the squaring operation of the result calculated by the ReLU activation function, b represents the bias vector, and A tt represents the attention matrix, ⊙ represents the dot product operation, and Avg represents average pooling over the frame dimension.

[0038] In this embodiment, the calculation method for the parameterless maximum value feature fusion operation and the parameterless minimum value feature fusion operation in step S4 includes:

[0039] A2 = Max(X) i )

[0040] A3 = Min(X) i )

[0041] Among them, X i This represents a batch of video frame-level embedding vectors generated by the visual encoder. Max indicates max pooling based on the frame dimension, and Min indicates min pooling based on the frame dimension.

[0042] In this embodiment, the multi-level cross-modal attention fusion operation calculation method in step S5 includes:

[0043] M=Adavg(concat(m1, m2, m3))

[0044]

[0045] V global =Sq(Split1(Att) out )+Split2(Att out )+Split3(Att out ))

[0046] In this context, `Adavg` represents adaptive pooling, `concat` represents concatenation along the frame dimension, `m1`, `m2`, and `m3` represent the results of expanding A1, A2, and A3 along the frame dimension, respectively, and `Conv` represents a one-dimensional convolution operation. This represents element-wise multiplication, σ represents the sigmoid activation function, and Att represents element-wise multiplication. out This represents the feature matrix after multi-level cross-modal attention fusion computation. Split1, Split2, and Split3 represent the feature matrix after Att is calculated. out It is split into three feature matrices according to the frame dimension, and Sq indicates compression according to the frame dimension.

[0047] In this embodiment, the method for calculating the cross-entropy loss function in step S6 includes:

[0048]

[0049] Where B represents the number of video category pairs, Let t represent the i-th global video-level embedding vector. i Let represent the i-th text embedding vector, cosine represents the cosine similarity calculation, and τ represents the temperature hyperparameter.

[0050] In this embodiment, the model structure of the CLIP-based video text embedding fusion classification method is as follows: Figure 2 As shown, its model training process includes:

[0051] Step A1: Obtain the videos to be classified and the corresponding classification labels for each video as training data, and divide the training data into training set and test set;

[0052] Step A2: Calculate the cross-entropy loss function for the training set according to the methods in steps S1 to S6;

[0053] Step A3: Adjust the parameters using the AdamW optimizer, and repeat step A2 until the training set loss value reaches its minimum and overfitting occurs.

[0054] Step A4: Input the test set into the trained CLIP-based video text embedding fusion classification model and output the predicted values ​​of the test set.

[0055] Step A5: When the test set predictions meet the expected final results, the trained CLIP-based video text embedding fusion classification method model is obtained.

[0056] We conducted experiments on two public datasets in the field of video action recognition, HMDB-51 and UCF-101.

[0057] During training, we set the batch size to 2, the gradient accumulation to 32, and the initial learning rate to 2×10. -6 The size of the dataset was determined, and k-shot datasets were created for both the HMDB-51 and UCF-101 datasets. K samples were randomly selected from each class for training, with k set to 2, 4, 8, and 16 respectively. The model was then tested using a test set. This setup was used to test the learning ability of the proposed classification method under limited supervision. To test the generalization ability of the proposed classification method to novel categories, the HMDB-51 and UCF-101 datasets were divided into a Base category and a Novel category, respectively. Training was performed on the Base category, followed by evaluation on both the Base and Novel categories. Figure 4 Experimental results show that the accuracy of this method on Top-1 is superior to other model methods.

[0058] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.< / y> < / y>

Claims

1. A CLIP-based video text embedding fusion classification method, which uses a classification model to process the video and outputs predicted values, characterized in that... The classification model is a CLIP-based video text embedding fusion classification model. The steps involved in constructing and processing a classification model include: Step S1: Build a text encoder and a visual encoder based on the pre-trained model CLIP to independently extract text features of labels and visual features at the video frame level, respectively. Step S2: Use a text encoder to independently encode the category label Y and wrap it in the prompt template to finally generate the text embedding vector t; Step S3, use a visual encoder to process video samples with T frames. Each image is independently encoded into a batch of images, and a batch of video frame-level embedding vector representations is generated. ; Step S4, for the embedding vector obtained in step S3 Perform the following operations respectively: Multi-scale adaptive feature fusion operation generates video-level embedding vectors , The parameterless maximum feature fusion operation generates video-level embedding vectors. , The parameterless minimum feature fusion operation generates video-level embedding vectors. ; Step S5, embed the video-level embedding vector generated in step S4. , and Multi-level cross-modal attention fusion operations are performed to ultimately generate global video-level embedding vectors. ; Step S6: Calculate the global video-level embedding vector generated in step S5. The maximum cosine similarity between the text embedding vector t and the text encoder and visual encoder of the CLIP model is calculated; and the text encoder and visual encoder of the CLIP model are fine-tuned by the cross-entropy loss function. The calculation method for the multi-scale adaptive feature fusion operation in step S4 includes: ， ， in, This represents a batch of video frame-level embedding vectors generated by the visual encoder. , and They represent The first learnable weight matrix, the second learnable weight matrix, and the third learnable weight matrix. , and These represent the first activation function, the second activation function, and the third activation function, respectively. , A transformation is an operation that shifts the components of a vector. This indicates that the result calculated using the ReLU activation function is squared. This represents the bias vector. Represents the attention matrix. This represents the dot product operation. This indicates that average pooling is performed on the frame dimension; The calculation method for the multi-level cross-modal attention fusion operation in step S5 includes: ， ， ， in, This indicates an adaptive pooling operation. This indicates that the frames are stitched together. They represent respectively to , and The result after expanding according to the frame dimension This represents a one-dimensional convolution operation. This represents element-wise multiplication. This represents the sigmoid activation function. This represents the feature matrix after multi-level cross-modal attention fusion computation. , and They respectively represent the following: The feature matrix is ​​split into three parts according to the frame dimension. This indicates compression based on the frame dimension.

2. The CLIP-based video text embedding fusion classification method according to claim 1, characterized in that, The calculation methods for the parameterless maximum value feature fusion operation and the parameterless minimum value feature fusion operation in step S4 include: ， ， in, This represents a batch of video frame-level embedding vectors generated by the visual encoder. This indicates that max pooling is performed according to the frame dimension. This indicates that minimum pooling is performed according to the frame dimension.

3. The CLIP-based video text embedding fusion classification method according to claim 1, characterized in that, The method for calculating the cross-entropy loss function in step S6 includes: ， Where B represents the number of video category pairs, This represents the i-th global video-level embedding vector. This represents the i-th text embedding vector. This indicates the calculation of cosine similarity. This indicates the temperature hyperparameter.

4. The CLIP-based video text embedding fusion classification method according to claim 1, characterized in that, The training process for a classification model includes: Step A1: Obtain the videos to be classified and the corresponding classification labels for each video as training data, and divide the training data into training set and test set; Step A2: Calculate the cross-entropy loss function for the training set according to the methods in steps S1 to S6; Step A3: Adjust the parameters using the AdamW optimizer, and repeat step A2 until the training set loss value reaches its minimum and overfitting occurs. Step A4: Input the test set into the trained classification model and output the predicted values ​​of the test set; Step A5: When the predicted values ​​on the test set meet the expected final effect, the trained classification model is obtained.

Images