A video text retrieval method based on BEiT-3 multimodal large model
Through the video text retrieval method based on the BEiT-3 model, the problem of insufficient generalization ability and accuracy of video text retrieval architecture in the existing technology is solved. The multi-grained cross-entropy loss function and self-attention mechanism are adopted to enhance the video text retrieval Performance, enabling more efficient video text retrieval.
Patent Information
- Application Number
- CN202410131751.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-31
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-01-31
AI Technical Summary
The existing video text retrieval architecture CLIP4Clip has shortcomings in generalization capabilities and accuracy, especially ignoring the timing and detailed information between video frames and text, resulting in a degradation in retrieval performance.
Using the video text retrieval method based on the BEiT-3 multimodal large model, the BEiT4clip model is constructed, the video and text are read, preprocessed and encoded, and the similarity between video frames and text is calculated, and the multi-grained cross entropy loss is used. 函数进行训练,结合自注意力机制和双向学习目标,增强视频文本检索的性能。
It improves the generalization ability and accuracy of video text retrieval, effectively enhances the model's performance in video text retrieval, and various similarity calculation methods improve retrieval efficiency.
Smart Images

Figure CN118377930B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of multimodal learning and discloses a video text retrieval method based on a BEiT-3 multimodal large model. Background Art
[0002] The BEiT-3 model is a general-purpose multimodal base model that has achieved advanced results in vision and vision-language tasks. The model uses a multi-way Transformer as the basic modeling architecture and can perform multiple tasks, including image-text retrieval tasks and general tasks. One of the key features of this model is the training method used when pre-training the multi-way Transformer. During the training phase, the image, text, and image-text pairs are masked, and then the original word units are restored through self-supervised learning. The multi-way Transformer consists of a shared self-attention module and a feedforward neural network. In the BEiT-3 model, a 40-layer multi-way Transformer is used. In the field of image-text retrieval tasks, BEiT-3 mainly uses the COCO and Flickr30K datasets for evaluation. When fine-tuning BEiT-3, it is used as a dual encoder for image-text retrieval work.
[0003] The dual encoder can extract features of images and texts respectively and calculate the cosine similarity value between them. In the fine-tuning stage, the BEiT-3 basic model trained with large-scale data was used. The pre-trained model was not trained using contrastive learning, but in the fine-tuning stage, contrastive learning was introduced as a loss function for training. Fewer steps were used in the fine-tuning process, achieving advanced results. In existing technologies, the BEiT-3 model has not been used in video text retrieval work.
[0004] The existing video retrieval architecture CLIP4Clip transfers the knowledge of the CLIP model to perform video text retrieval. The model is trained to calculate the similarity between video and text by learning a function, and then sorts the videos or texts for the given query video based on the calculated similarity. This method simply uses the CLIP model, and the generalization ability of video text retrieval is low. At the same time, it only uses a single-granularity loss function, which may ignore the timing and detail information between video frames and text, resulting in reduced retrieval performance. Summary of the invention
[0005] Purpose of the invention: To provide a video text retrieval method based on the BEiT-3 multimodal large model to solve the above-mentioned problems existing in the prior art.
[0006] Technical solution: A video text retrieval method based on the BEiT-3 multimodal large model, comprising the following steps:
[0007] S1. Build the BEiT4clip video text retrieval model, read the video and text, pre-process the video and text, and obtain the video frame input representation and text input representation;
[0008] S2, using the BEiT-3 encoder to encode the video frame input representation and the text input representation to obtain video features and text features;
[0009] S3, performing average pooling on the video features to obtain average frame features, and calculating the similarity between the video frame and the text based on the average frame features and the text features;
[0010] S4. Based on the similarity between video frames and texts, a loss function is constructed to train the BEiT4clip video text retrieval model to realize video text retrieval.
[0011] According to one aspect of the present application, step S1 specifically comprises:
[0012] S11, constructing a BEiT4clip video text retrieval model, reading a video, and dividing the video into a predetermined number of video frames;
[0013] S12, hierarchically downsampling the video frame, dividing the downsampled video frame through a sliding window to obtain a predetermined number of image blocks, projecting the image block using a linear layer to obtain an embedding vector of the image block;
[0014] S13, forming a video frame input representation based on the embedding vector of the image block by using the label, position encoding and type encoding;
[0015] S14, reading the text, dividing the text into a predetermined number of text words, and converting the text words into embedding vectors of the text words using a word embedding matrix;
[0016] S15. Based on the embedding vector of text words, position encoding and type encoding are used to form a text input representation.
[0017] According to one aspect of the present application, step S2 is specifically:
[0018] S21, using the BEiT-3 image encoder to encode the video frame input representation to obtain video features;
[0019] S22. Use the BEiT-3 text encoder to encode the text input representation to obtain text features.
[0020] According to one aspect of the present application, step S3 is specifically:
[0021] S31, obtaining time series features of video frames through a Transformer encoder, embedding the time series features into video features to obtain video frame features, and performing average pooling on the video frame features to obtain average frame features;
[0022] S32, based on the text features, using a tagging method to form a text representation;
[0023] S33. Normalize the average frame features and text representation, then linearly project them into the multimodal embedding space, construct a similarity function, and calculate the similarity between the video frame and the text.
[0024] According to one aspect of the present application, step S4 is specifically:
[0025] S41, using softmax operation to normalize the similarity between the video frame and the text to obtain a probability distribution;
[0026] S42. Based on the probability distribution, a symmetric cross entropy loss function is constructed, and the symmetric cross entropy loss function is optimized by using a gradient descent algorithm, and the parameters of the BEiT4clip video text retrieval model are updated to obtain a trained BEiT4clip video text retrieval model;
[0027] S44, execute the trained BEiT4clip video text retrieval model to retrieve the video text.
[0028] According to one aspect of the present application, step S33 also includes training the similarity between the video frame and the text, specifically:
[0029] S33a, using GPT-2 to generate descriptive text for the video, calculating the distance between the video frame and the descriptive text through the CLIP model, and optimizing the generated descriptive text;
[0030] S33b, preprocessing the optimized descriptive text, using a text encoder to convert the preprocessed descriptive text into subtitle coding features;
[0031] S33c. Based on the subtitle encoding features, a two-way learning objective method is adopted to construct a cross entropy loss function of text and subtitles to train the similarity between video frames and text.
[0032] According to one aspect of the present application, step S21 further includes:
[0033] S21a, based on the video frame input representation, obtaining a predetermined number of time RGB values, and superimposing them to obtain differential features;
[0034] S21b, average pooling the differential features, and using a two-dimensional convolutional neural network to extract features from the average pooled differential features to obtain convolution features;
[0035] S21c, using the pre-trained residual network to encode the convolutional features to obtain encoded features;
[0036] S21d, fusing the coding features and video features through weights to obtain temporal and spatial features.
[0037] According to one aspect of the present application, step S3 further includes:
[0038] S3a, using a text encoder to obtain the representation of each word in the text feature, and based on the video feature and the representation of each word, using matrix multiplication to calculate the similarity between the video frame and each word in the text;
[0039] S3b, using the BEiT-3 video encoder to obtain the features of each frame in the video features, and based on the text features and the features of each frame, using matrix multiplication to calculate the similarity between the text and each frame in the video;
[0040] S3c, based on the representation of each word in the text feature and the features of each frame in the video feature, the similarity between each frame in the video and each word in the text is calculated using matrix multiplication;
[0041] S3d, using the self-attention mechanism, performs weighted aggregation on the similarity between the video frame and each word in the text, the similarity between the text and each frame in the video, and the similarity between each frame in the video and each word in the text to obtain the fine-grained similarity between the video frame and the text.
[0042] According to one aspect of the present application, step S42 further includes:
[0043] S42a, constructing a multi-granularity cross entropy loss function based on the similarity between the video frame and the text and the similarity between the fine-grained video frame and the text;
[0044] S42b, constructing a total loss function based on the multi-granularity cross entropy loss function and the cross entropy loss function of text and subtitles;
[0045] S42c. Based on the total loss function, update the parameters of the BEiT4clip video text retrieval model to obtain a trained BEiT4clip video text retrieval model.
[0046] Beneficial effects: The present invention successfully applies the BEiT-3 model to video text retrieval, improves the generalization ability of video text retrieval, and simultaneously conducts multiple explorations on the similarity calculation between video frames and texts, thereby improving the accuracy and efficiency of video text retrieval, and effectively enhancing the performance of the model in video text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Figure 1 It is a flow chart of the present invention.
[0048] Figure 2 This is a flow chart of step S1 of the present invention.
[0049] Figure 3 This is a flow chart of step S2 of the present invention.
[0050] Figure 4 This is a flow chart of step S3 of the present invention.
[0051] Figure 5 This is a flow chart of step S4 of the present invention.
[0052] Figure 6 This is a model diagram of the present invention.
[0053] Figure 7 This is a diagram of the BEiT-3 model of the present invention. DETAILED DESCRIPTION
[0054] like Figure 1 As shown, the BEiT-3 large model of the present invention is fine-tuned to the field of video retrieval, and the similarity between the video and the text is calculated as training. A video text retrieval algorithm BEiT4clip based on the BEiT-3 multimodal large model is proposed, which is specifically:
[0055] S1. Build the BEiT4clip video text retrieval model, read the video and text, pre-process the video and text, and obtain the video frame input representation and text input representation;
[0056] like Figure 2 As shown, step S1 is specifically as follows:
[0057] S11, constructing a BEiT4clip video text retrieval model, reading a video, and dividing the video into a predetermined number of video frames;
[0058] S12, hierarchically downsampling the video frame, dividing the downsampled video frame through a sliding window to obtain a predetermined number of image blocks, projecting the image block using a linear layer to obtain an embedding vector of the image block;
[0059] S13, forming a video frame input representation based on the embedding vector of the image block by using the label, position encoding and type encoding;
[0060] S14, reading the text, dividing the text into a predetermined number of text words, and converting the text words into embedding vectors of the text words using a word embedding matrix;
[0061] S15. Based on the embedding vector of text words, position encoding and type encoding are used to form a text input representation.
[0062] The video frame is hierarchically downsampled to obtain the features of the initial video frame, such as obtaining a feature map that is downsampled by 16 times. Then, the downsampled video frame is segmented into several non-overlapping image blocks, such as 16x16 pixels, through a sliding window. Then, a linear layer is used to project them into a one-dimensional space to obtain the embedding vector of the image block.
[0063] In a further embodiment, non-overlapping portions of the video frame are sliced, expanded into a plane vector and linearly projected into a one-dimensional space to obtain an image block embedding, a learnable label [V_CLS] is added to the image block embedding, and finally a learnable one-dimensional position embedding V is obtained through the image block embedding. pos ∈R (N+1)×D and image type embedded V type ∈R D , the input represented by the video frame is obtained by the following summation:
[0064]
[0065] Among them, H o v ∈R (N+1)×D , the linear projection dimension is defined as , Vv i p represents the image patch embedding, v [V_CLS] represents the learnable mark, V pos represents the learnable position embedding, V type represents the image type embedding. Then the input representation of the entire video sequence frame can be expressed as a matrix For text representation, the start marker [T_CLS] and special boundary marker [T_SEP] of the sequence are added to the text sequence. Text input representation is represented by word embedding, text position embedding and text type embedding, specifically:
[0066]
[0067] where w [T_CLS] Indicates the start mark of the sequence, w i represents word embedding, w[T_SEP] A special boundary marker representing a sequence, T pos represents the position embedding of the text, T type A type embedding representing text.
[0068] S2, using the BEiT-3 encoder to encode the video frame input representation and the text input representation to obtain video features and text features;
[0069] like Figure 3 As shown, step S2 is specifically as follows:
[0070] S21, using the BEiT-3 image encoder to encode the video frame input representation to obtain video features;
[0071] S22. Use the BEiT-3 text encoder to encode the text input representation to obtain text features.
[0072] In a further embodiment, the features of video and text are encoded by the BEiT-3 model. Since BEiT-3 is currently only used for feature extraction of images and text, it is necessary to first migrate the model to feature extraction of video and text. In this step, video frames are extracted from the edited video, and then the features of the video frames are encoded by the image feature encoder corresponding to BEiT-3. The encoder used is a multi-channel Transformer. Given the output vector H of the previous layer, l-1 , l ∈[1,L], then each multi-way Transformer block will capture features by switching to different modal experts and align visual and language content through multi-head self-attention shared across modalities.
[0073] Among them, MSA stands for multi-head self-attention, LN stands for layer normalization, MoME~FFN stands for multimodal experts, and H l ' represents the output vector after multi-head self-attention processing, H l It represents the features after being encoded by modality experts. Different experts are used to encode different input contents. When obtaining video features, visual experts are used to encode the content of the video frame. When obtaining text features, language experts are used to encode the text.
[0074] S3, performing average pooling on the video features to obtain average frame features, and calculating the similarity between the video frame and the text based on the average frame features and the text features;
[0075] like Figure 4 As shown, step S3 is specifically as follows:
[0076] S31, obtaining time series features of video frames through a Transformer encoder, embedding the time series features into video features to obtain video frame features, and performing average pooling on the video frame features to obtain average frame features;
[0077] S32, based on the text features, using a tagging method to form a text representation;
[0078] S33. Normalize the average frame features and text representation, then linearly project them into the multimodal embedding space, construct a similarity function, and calculate the similarity between the video frame and the text.
[0079] In a further embodiment, the time series features are embedded into the video features using the Transformer encoder, and then all the video frame features are aggregated by average pooling to obtain the average frame representation, which is then normalized with the text representation and linearly projected into the multimodal embedding space, and then the similarity between the video frame and the text is calculated. The average frame representation is defined as:
[0080]
[0081] Among them, mean_pooling(.,.) represents the average pooling operation, H i v Represents video features.
[0082] Then the similarity function is defined as s(v i , t i ), which is calculated as follows:
[0083]
[0084] in Represents text features, Represents video features.
[0085] S4. Based on the similarity between video frames and texts, a loss function is constructed to train the BEiT4clip video text retrieval model to realize video text retrieval.
[0086] like Figure 5 As shown, step S4 is specifically as follows:
[0087] S41, using softmax operation to normalize the similarity between the video frame and the text to obtain a probability distribution;
[0088] S42. Based on the probability distribution, a symmetric cross entropy loss function is constructed, and the symmetric cross entropy loss function is optimized by using a gradient descent algorithm, and the BEiT4clip video text retrieval model is updated to obtain a trained BEiT4clip video text retrieval model;
[0089] S44, execute the trained BEiT4clip video text retrieval model to retrieve the video text.
[0090] In a further embodiment, the similarity between the obtained video frame and the text is used to design a loss function to train the model using the value obtained by similarity calculation. The obtained value is used to train the model parameters using a symmetric cross entropy loss.
[0091]
[0092] Where L v2t is the cross entropy loss value of retrieving text from video, L t2v is the cross entropy loss value of video retrieval through text. The total loss value L is obtained by calculating the sum of the loss values. B represents the number of data processed in a batch, and v i Represents the characteristics of the video, t i Represents the characteristics of the text, s(v i , t i ) represents the similarity between video and text.
[0093] Furthermore, the present invention fine-tunes the BEiT-3 model to achieve superior performance in the video text retrieval task. The BEiT-3 pre-trained model used is beit3_large_itc_patch16_224, which is fine-tuned for image text retrieval on the COCO dataset and Flickr30k. The base model is not trained using contrastive learning when it is not fine-tuned, but by randomly masking a certain proportion of text tags and image patches, using self-supervised learning to restore the original word unit in the case of input corruption.
[0094] In the fine-tuning stage, the model is fine-tuned to a dual encoder to encode images and texts separately. A small amount of contrastive learning loss is introduced during training to achieve advanced results. The advantage here is that the base model is trained with a large amount of data. Compared with the CLIP model, the BEiT-3 model has significantly enhanced text and image encoding capabilities. Considering that many tasks are studied based on CLIP, it is more reasonable to choose the more advanced BEiT-3 model as the encoder for video tasks.
[0095] After obtaining the image and text encoders, to prevent overfitting, BEiT4clip freezes the first 3 / 4 layers of the BEiT-3 image and text encoder parts during fine-tuning. The encoder module consists of a self-attention sharing module and a feedforward neural network for images. The BEiT-3 model contains a 40-layer multi-way Transformer, and BEiT4clip chooses to retain 10 layers for fine-tuning on the MSRVTT dataset. The frozen layers have obtained good results in the BEiT-3 base model training, and the fine-tuned layers can further learn to adapt to specific tasks. In the fine-tuning strategy, symmetric cross entropy loss is introduced for fine-tuning to further optimize the performance of the model in the video text retrieval task.
[0096] In a further embodiment, the innovative BEiT-3 model pre-training is introduced in the feature extraction stage of video and text retrieval. Its main method flow and technical points are different from the current method that mainly uses the pre-trained CLIP model. The main steps are:
[0097] Step 1: By slicing and mapping the video frame, the embedding of the video frame is obtained to form the video frame input representation. At the same time, by marking and segmenting the text words, the text embedding representation is obtained to obtain the text input representation.
[0098] Step 2: Input the above input representation into the corresponding encoder of the BEiT-3 model to obtain the encoding features. The pre-training knowledge of the BEiT-3 model for images and texts will be applied to the feature extraction process of videos and texts.
[0099] Step 3: Use the Transformer encoder to embed the temporal features into the video features. Then, the information of all video frames is aggregated through average pooling to obtain the representation of the average frame. Next, the representation is normalized with the representation of the text and linearly projected into the multimodal embedding space. Finally, the similarity between the video frame and the text is calculated.
[0100] Step 4: Using the similarity between the video and the text obtained above, design a loss function to train the model. The model training is completed through the value of similarity calculation to achieve video text retrieval.
[0101] By introducing the BEiT-3 model, this paper makes the feature extraction stage in the video and text retrieval task more innovative. The entire process makes full use of the image and text encoding capabilities of the BEiT-3 model and trains through similarity calculation, thereby improving the effect of video text retrieval.
[0102] Further, such as Figure 6As shown in the figure, after obtaining text and video data, they are first preprocessed so that the processed data can be input into the multi-way Transformer for feature encoding. For text data, learnable labels are introduced and input into the multi-way Transformer as word embedding representations. A feedforward neural network of a language expert is selected to learn feature representations for text.
[0103] For video data, we first extract video frames, expand them into two-dimensional image patches, and then project them into a one-dimensional embedding sequence, which is fed into a multi-way Transformer as input for learnable labels. For video data, a feedforward neural network from a visual expert is used to learn the feature representation of video frames.
[0104] The Transformer encoder processes the time series relationship between video frames and embeds the time series features into the video features. Then, the average pooling layer averages the information of all video frames to obtain the information of an average frame. This average frame information contains the time information and the feature information of the video frame.
[0105] Finally, the obtained video features and text features are subjected to multi-granular similarity calculation to obtain similarity values. Through this series of processing steps, the system can comprehensively consider the information of text and video and provide more effective feature representation for subsequent video text retrieval tasks.
[0106] like Figure 7 As shown in Figure 1, the present invention uses a fine-tuned version of the BEiT-3 model, namely beit3_large_itc_patch16_224, as the encoder for images and texts. By extracting video frames, the knowledge pre-trained by BEiT-3 in the field of image text is successfully applied to the retrieval task of video text.
[0107] During the model training process, self-supervised learning was used to construct the optimization target. Since the BEiT-3 large model has excellent performance in processing the semantic relationship between image and text features, the advanced effect of the task is achieved by fine-tuning the video text retrieval task. By introducing a contrastive learning training strategy, the present invention aims to maximize the use of BEiT-3's advanced knowledge in the fields of images and text to improve the model's adaptability and performance for video text retrieval tasks. This innovative method is expected to bring significant improvements to the field of video text retrieval and provide valuable practical experience for related research. Previously, BEiT-3 was invented and used in the field of multimodal learning of images and this article, and no work has used it in video text retrieval.
[0108] By using the more powerful BEiT-3 model as the feature encoder for video and text, the recall rate of video and text retrieval has been significantly improved. This is the first time that the BEiT-3 model has been successfully applied to video tasks, successfully transferring the knowledge learned by BEiT-3 on images and text to video text retrieval.
[0109] By fine-tuning on the video text retrieval task and introducing symmetric cross entropy loss as the task loss for training, the model's performance in video text retrieval is effectively enhanced. This innovative method not only demonstrates the superior performance of the BEiT-3 model in the image and text fields, but also brings new possibilities for its application in the video field. The powerful feature encoding capability of the model plays a significant role in improving the video text retrieval effect. This invention lays the foundation for further exploration of video and text tasks based on the BEiT-3 model, and provides a valuable reference for future research in related fields of multimodal learning.
[0110] Furthermore, step S33 also includes training the similarity between the video frame and the text, specifically:
[0111] S33a, using GPT-2 to generate descriptive text for the video, calculating the distance between the video frame and the descriptive text through the CLIP model, and optimizing the generated descriptive text;
[0112] S33b, preprocessing the optimized descriptive text, using a text encoder to convert the preprocessed descriptive text into subtitle coding features;
[0113] S33c. Based on the subtitle encoding features, a two-way learning objective method is adopted to construct a cross entropy loss function of text and subtitles to train the similarity between video frames and text.
[0114] In a further embodiment, auxiliary subtitles are generated for video text similarity training. The knowledge in the large language model is used to generate subtitles for the video, and the next word is predicted from the initial prompt through GPT-2. Then, the CLIP loss for calibration is used to drive the model to generate sentences describing the video, and the generated description sentences are used to represent the video to achieve text-to-text retrieval, and the training is enhanced by calculating the cosine similarity between the features of the generated text and the features of the query text.
[0115] The similarity between the generated subtitles and the text is calculated. By considering the bidirectional learning objective, the stacked cross entropy loss is used to maximize the similarity between the matching query text and the generated subtitle pairs, and minimize the similarity between other query text and generated subtitle pairs.
[0116]
[0117] Among them, s qc ( ) represents the similarity function between the query text and the generated text, e ti represents the embedding features of the query text, e ci represents the embedding features of the generated subtitles, and τ represents the hyperparameter used for scaling, L Q2C represents the cross entropy loss value of query text retrieval subtitles, L C2Q represents the cross entropy loss value of the subtitle retrieval query text, L QC Represents the total loss between the query text and the generated captions.
[0118] Furthermore, step S21 further includes:
[0119] S21a, based on the video frame input representation, obtaining a predetermined number of time RGB values, and superimposing them to obtain differential features;
[0120] S21b, average pooling the differential features, and using a two-dimensional convolutional neural network to extract features from the average pooled differential features to obtain convolution features;
[0121] S21c, using the pre-trained residual network to encode the convolutional features to obtain encoded features;
[0122] S21d, fusing the coding features and video features through weights to obtain temporal and spatial features.
[0123] In a further embodiment, the feature encoding of the video time information is performed. Time information is an important feature that distinguishes video from images. The time information of the video is extracted by time difference, and then fused with the video features to obtain the time-space feature, so that the video frame can perceive the local movement. The information of the time feature is conducive to the model to better understand the previous and next information of the video and enhance the semantic understanding of the text.
[0124] First, extract the video time difference information. For the input video frame, extract several time RGB values in the local window centered on the frame, and then i )=[D -2 ,D -1 ,D1,D2] are superimposed to obtain the video frame difference information.
[0125] Then, the temporal difference information is encoded. The obtained RGB difference is halved using average pooling, and the features after average pooling are convolved using a two-dimensional convolutional neural network. The obtained features are then encoded through a pre-trained residual network to obtain the encoded features of the temporal difference information.
[0126] Finally, the temporal difference feature is fused with the video feature. Different weights are given to the temporal difference feature and the time feature through linear weighting, and the temporal difference feature and the video feature are fused. For example, the weight given to the video feature is 0.8, and the weight given to the temporal difference feature is 0.2. The two features are added together to obtain the video feature that integrates the temporal difference feature information.
[0127] Furthermore, step S3 also includes:
[0128] S3a, using a text encoder to obtain the representation of each word in the text feature, and based on the video feature and the representation of each word, using matrix multiplication to calculate the similarity between the video frame and each word in the text;
[0129] S3b, using the BEiT-3 video encoder to obtain the features of each frame in the video features, and based on the text features and the features of each frame, using matrix multiplication to calculate the similarity between the text and each frame in the video;
[0130] S3c, based on the representation of each word in the text feature and the features of each frame in the video feature, the similarity between each frame in the video and each word in the text is calculated using matrix multiplication;
[0131] S3d, using the self-attention mechanism, performs weighted aggregation on the similarity between the video frame and each word in the text, the similarity between the text and each frame in the video, and the similarity between each frame in the video and each word in the text to obtain the fine-grained similarity between the video frame and the text.
[0132] In a further embodiment, multi-granularity matching is performed between video and text. Currently, only coarse-grained matching between video and text is considered for similarity calculation between video and text, but fine-grained matching is missing. Therefore, a matching situation between the fine-grained frames in the video and the words in the text is considered, and the similarity between the fine-grained matching is calculated, thereby improving the effect of video text retrieval.
[0133] In the previous comparison between videos and sentences, after obtaining the video features, we obtained an average feature v'∈R of the entire video through average pooling dim , obtain the text feature t'∈R through the text encoder dim , these two features belong to coarse-grained features, where R dim Represents the feature dimension of the data. Then matrix multiplication is used to evaluate the similarity between the video and the sentence: S V-S =(v' ) T (t'). This formula represents the similarity score between the video and the sentence.
[0134] Now compare the video with the word and use the text encoder Transformer to obtain the word representation vector T∈R m×dim The video representation v'∈R obtained by the video encoder dim , calculate the similarity between the video representation and each word representation through matrix multiplication: S V-W =(T v ' ) T . This formula represents the similarity score between the video and the word.
[0135] Compare sentences with frames and use the features of video frames obtained by BEiT-3’s video encoder The sentence representation t'∈R obtained by the text encoder dim , using matrix multiplication to express the similarity score between the two, the formula is: .
[0136] Compare the frames with the words and the features of the video frames obtained by the video encoder And the word features T∈R obtained from the text encoding words m×dim , using matrix multiplication to represent the fine-grained similarity matrix between them: , where S F-S ∈R n×m is a fine-grained similarity matrix, where n and m represent the number of frames and words, respectively.
[0137] Finally, the self-attention mechanism is used to assign weights to fine-grained features and calculate similarity scores. The Softmax operation is used to obtain the weight of the similarity vector, and then self-attention is used to assign different weights to the similarity matrix during the aggregation process to increase the weight of the fine-grained features related to the query:
[0138]
[0139] Among them, S V-W ∈R 1×m is the similarity between the video and the word, S F-S ∈R n×1 is the similarity between the video frame and the sentence, S' V-W It represents the similarity between the video and the word after the weight is given, S' F-S It represents the similarity between the video frame and the sentence after the weight is given, and τ represents the temperature parameter of the Softmax operation.
[0140] Considering the fine-grained matrix S F-W ∈R n×mcontains the similarity scores between n frames and m words, so two attention operations are performed on the matrix to obtain the instance-level similarity vector of the fine-grained matrix. First, the fine-grained video-level and sentence-level similarity vectors are obtained:
[0141]
[0142] Among them, * represents all the contents in the dimension, S vid ∈R 1×m It represents the similarity score between the video and the m words in the sentence, S sen ∈R n×1 It represents the similarity score between the sentence and n frames in the video.
[0143] Then a second attention operation is performed to obtain the instance-level similarity score:
[0144]
[0145] Where S' vid ∈R 1 and S' sen ∈R are all instance-level similarities, and then the similarities are averaged to obtain the fine-grained similarity scores between frames and words:
[0146]
[0147] Furthermore, step S42 further includes:
[0148] S42a, constructing a multi-granularity cross entropy loss function based on the similarity between the video frame and the text and the similarity between the fine-grained video frame and the text;
[0149] S42b, constructing a total loss function based on the multi-granularity cross entropy loss function and the cross entropy loss function of text and subtitles;
[0150] S42c. Based on the total loss function, update the parameters of the BEiT4clip video text retrieval model to obtain a trained BEiT4clip video text retrieval model.
[0151] In a further embodiment, the similarity scores between multiple granularities are averaged to obtain a similarity score including multiple granularities:
[0152]
[0153] The retrieval model is then optimized by using a symmetric cross entropy loss on this similarity matrix to obtain an L1 loss:
[0154]
[0155] Where L v2t represents the cross entropy loss when retrieving text from video, L t2v It represents the cross entropy loss when retrieving videos through text, and the cross entropy loss function L obtained before between the query text and the generated subtitles is QC Summing with L1 gives the total loss function L, which is used to optimize the model:
[0156] L = L1 + L QC .
[0157] The present invention successfully applies the BEiT-3 model to video text retrieval, improves the generalization ability of video text retrieval, and simultaneously conducts multiple explorations on the similarity calculation between video frames and texts, thereby improving the accuracy and efficiency of video text retrieval and effectively enhancing the performance of the model in video text retrieval.
[0158] The preferred embodiments of the present invention are described in detail above; however, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and these equivalent transformations all belong to the protection scope of the present invention.
Claims
1. A video text retrieval method based on the BEiT-3 multimodal large model, characterized in that: The steps include: S1. Build the BEiT4clip video text retrieval model, read the video and text, pre-process the video and text, and obtain the video frame input representation and text input representation; S2, using the BEiT-3 encoder to encode the video frame input representation and the text input representation to obtain video features and text features; S3, performing average pooling on the video features to obtain average frame features, and calculating the similarity between the video frame and the text based on the average frame features and the text features; S4. Based on the similarity between video frames and texts, a loss function is constructed to train the BEiT4clip video text retrieval model to realize video text retrieval; Step S1 is specifically as follows: S11, constructing a BEiT4clip video text retrieval model, reading a video, and dividing the video into a predetermined number of video frames; S12, hierarchically downsampling the video frame, dividing the downsampled video frame through a sliding window to obtain a predetermined number of image blocks, projecting the image block using a linear layer to obtain an embedding vector of the image block; S13, forming a video frame input representation based on the embedding vector of the image block by using the label, position encoding and type encoding; S14, reading the text, dividing the text into a predetermined number of text words, and converting the text words into embedding vectors of the text words using a word embedding matrix; S15, based on the embedding vector of the text words, position encoding and type encoding are used to form a text input representation; Step S2 is specifically as follows: S21, using the BEiT-3 image encoder to encode the video frame input representation to obtain video features; S22, using the BEiT-3 text encoder to encode the text input representation to obtain text features; The step S2 further comprises: Step S23: extract the time information of the video using time difference, and then fuse it with the video features to obtain time-space features, so that the video frame can perceive local motion. The information of time features is conducive to the model to better understand the previous and next information of the video and enhance the semantic understanding of the text; Step S231, extracting video time difference information; For the input video frame, several temporal RGB values are extracted from the local window centered on the frame, and then superimposed along each channel to obtain the video frame difference information; Step S232, encoding the time difference information; Use average pooling to halve the obtained RGB difference, use a two-dimensional convolutional neural network to perform a convolution operation on the features after average pooling, and then encode the obtained features through a pre-trained residual network to obtain the encoding features of the temporal difference information; Step S233, fusing the time difference feature with the video feature; Different weights are given to the temporal difference feature and the time feature through linear weighting, and the temporal difference feature is fused with the video feature to obtain the video feature that integrates the temporal difference feature information.
2. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 1 is characterized in that: Step S3 is specifically as follows: S31, obtaining time series features of video frames through a Transformer encoder, embedding the time series features into video features to obtain video frame features, and performing average pooling on the video frame features to obtain average frame features; S32, based on the text features, using a tagging method to form a text representation; S33. Normalize the average frame features and text representation, then linearly project them into the multimodal embedding space, construct a similarity function, and calculate the similarity between the video frame and the text.
3. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 2 is characterized in that: Step S4 is specifically as follows: S41, using softmax operation to normalize the similarity between the video frame and the text to obtain a probability distribution; S42. Based on the probability distribution, a symmetric cross entropy loss function is constructed, and the symmetric cross entropy loss function is optimized by using a gradient descent algorithm, and the parameters of the BEiT4clip video text retrieval model are updated to obtain a trained BEiT4clip video text retrieval model; S43. Execute the trained BEiT4clip video text retrieval model to retrieve the video text.
4. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 3 is characterized in that: Step S33 also includes training the similarity between the video frame and the text, specifically: S33a, using GPT-2 to generate descriptive text for the video, and using the CLIP model to calculate the distance between the video frame and the descriptive text, and optimize the generated descriptive text; S33b, preprocessing the optimized descriptive text, using a text encoder to convert the preprocessed descriptive text into subtitle encoding features; first, generating a descriptive sentence of the video through GPT-2, then using the CLIP model to calculate the distance between the video frame and the generated sentence, while GPT-2 is inferring, optimizing the model through CLIP, so that a description that meets the video requirements can be generated; finally, preprocessing the generated subtitles, inputting them into the text encoder to generate subtitle encoding features, and the encoder used is the BEiT-3 pre-trained text encoder; S33c. Based on the subtitle encoding features, a two-way learning objective method is adopted to construct a cross entropy loss function of text and subtitles to train the similarity between video frames and text.
5. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 4 is characterized in that: Step S21 also includes: S21a, based on the video frame input representation, obtaining a predetermined number of time RGB values, and superimposing them to obtain differential features; S21b, average pooling the differential features, and using a two-dimensional convolutional neural network to extract features from the average pooled differential features to obtain convolution features; S21c, using the pre-trained residual network to encode the convolutional features to obtain encoded features; S21d, fusing the coding features and video features through weights to obtain temporal and spatial features.
6. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 5 is characterized in that: Step S3 also includes: S3a, using a text encoder to obtain the representation of each word in the text feature, and based on the video feature and the representation of each word, using matrix multiplication to calculate the similarity between the video frame and each word in the text; S3b, using the BEiT-3 video encoder to obtain the features of each frame in the video features, and based on the text features and the features of each frame, using matrix multiplication to calculate the similarity between the text and each frame in the video; S3c, based on the representation of each word in the text feature and the features of each frame in the video feature, the similarity between each frame in the video and each word in the text is calculated using matrix multiplication; S3d, using the self-attention mechanism, performs weighted aggregation on the similarity between the video frame and each word in the text, the similarity between the text and each frame in the video, and the similarity between each frame in the video and each word in the text to obtain the fine-grained similarity between the video frame and the text.
7. The video text retrieval method based on the BEiT-3 multimodal large model according to claim 6 is characterized in that: Step S42 also includes: S42a, constructing a multi-granularity cross entropy loss function based on the similarity between the video frame and the text, and the similarity between the fine-grained video frame and the text; S42b, constructing a total loss function based on the multi-granularity cross entropy loss function and the cross entropy loss function of text and subtitles; S42c. Based on the total loss function, update the parameters of the BEiT4clip video text retrieval model to obtain a trained BEiT4clip video text retrieval model.
Citation Information
Patent Citations
End-to-end multi-granularity contrast learning method for video text retrieval
CN115757713A
Unsupervised video description method based on self-constrained dynamic text features
CN116109978A
Video text retrieval method based on differential multi-scale multi-granularity feature fusion
CN116226449A
Cross-modal video text retrieval method, system and equipment and medium
CN116910307A
Method and device for determining similarity between text and video
CN116958868A
Cited By
Multi-label cross-modal video-text retrieval method based on Mamba and storage medium
CN120780871A
