Video-text retrieval method based on multi-modal large model

Abstract

The invention discloses a video-text retrieval method based on a multi-modal large model. Comprising the following steps: acquiring an original text-original video pair data set; generating a supplementary text and an enhanced video for the original video by using a preset method; calculating fusion text features of the original text and the supplementary text, and respectively extracting video global features of the original video and the enhanced video by using cross attention coding; calculating a contrast learning constraint by using the text features and the video global features; promoting a model to realize information alignment by utilizing a comparative learning constraint; and generating feature information of a to-be-retrieved video by using the trained model, storing the feature information, and obtaining a retrieval result by calculating the similarity between the features of the input video or text and the stored feature information during retrieval. According to the method, through data enhancement based on the multi-modal large model and tetrad video-text comparative learning, the video and the text are fully aligned, and high recall precision of video-text retrieval is obtained.

Description

Technical Field

[0001] The present invention belongs to the field of video retrieval technology, and more specifically, relates to a video-text retrieval method based on a multimodal large model. Background Art

[0002] Currently, video-text retrieval is mainly based on the image-text contrast learning framework and its derivative frameworks. Directly applying such frameworks to the video-text field does not take into account the large information bias between the video modality and the text modality, which is mainly reflected in the fact that the text often only describes part of the content in the video, or the text is a highly abstract summary of the video content. In order to compensate for the information bias, previous methods often rely on a large amount of data for secondary training, which requires a lot of time for manual annotation and additional training costs.

[0003] The invention patent with the prior art publication number CN114817627A proposes a cross-modal retrieval method from text to video based on multi-faceted video representation learning, including: obtaining preliminary features of video and text; using a video storyboard tool to group the initial video frames according to different scenes, and then inputting the display coding branch for explicit coding to obtain explicit multi-faceted representations of different scenes of the video; inputting the initial video features into the implicit coding branch, implicitly encoding the initial video features through the leading feature multi-attention network, and obtaining implicit multi-faceted representations expressing different semantic contents of the video; fusing the multi-faceted encodings of the two branches to obtain a multi-faceted video feature representation; mapping the multi-faceted video feature representation and the text feature to the common space respectively, using the common space learning algorithm to learn the correlation between the two modalities, and training the model in an end-to-end manner to achieve cross-modal retrieval from text to video. This scheme only uses the original text, resulting in a limited space for semantic expression, which is prone to semantic differences and makes the text unable to fully cover the video information. In addition, the scheme focuses too much on cross-modal alignment and ignores the alignment within the modality. Summary of the invention

[0004] The present invention aims to solve the problem in the prior art that the text semantic space is limited, cross-modal alignment is overly focused on while intra-modal alignment is neglected, and a video-text retrieval method based on a multimodal large model is provided. The method comprises the following steps: Get the original text-original video pair dataset; Generate supplementary text and enhanced video for original video using preset methods; Calculate the fused text features of the original text and the supplemented text, and use cross-attention encoding to extract the global video features of the original video and the enhanced video respectively; Calculating contrastive learning constraints using the text features and the video global features; Using the contrastive learning constraint to promote the model to achieve information alignment; The trained model is used to generate feature information of the video to be retrieved and saved. During retrieval, the similarity between the features of the input video or text and the saved feature information is calculated to obtain the retrieval results.

[0005] Furthermore, the method for generating supplementary text is to use a multimodal large language model to generate a detailed text description for the original video.

[0006] Furthermore, the method for generating enhanced video includes the following steps: sampling frames of the original video; applying image enhancement technology to the sampled video frames; and recombining the processed frame sequence in order to generate enhanced video.

[0007] Furthermore, the image enhancement technology includes at least one of the following: rotation, flipping, color transformation, space transformation, style transformation, and geometric transformation of the original video.

[0008] Furthermore, the specific method of calculating the fusion text features is: Use a word segmenter to segment the original text and the supplementary text into discrete words respectively; Mapping discrete words into continuous low-dimensional vectors through an embedding layer; Inputting the low-dimensional vector into a deep learning model to extract original text features and supplementary text features; The fused text features are obtained by summing the original text features and the supplementary text features.

[0009] Furthermore, the specific method for extracting the global features of the video is: Sampling video frames from a video; Split each video frame into non-overlapping image blocks; Perform convolution operations on each image block to generate a low-dimensional vector and add position encoding and category labels; The low-dimensional vector with position encoding and category label is input into the video deep learning model, and the obtained category label is used as the feature vector of each video frame; The global features of the video are calculated by fusion text features and feature vectors of each video frame using cross attention encoding.

[0010] Furthermore, the contrastive learning constraints include contrastive learning constraints between text and video across modalities, and global feature constraints between original video and enhanced video within a modality.

[0011] Furthermore, the contrast learning constraints between text and video include original text-original video contrast learning constraints and original text-enhanced video contrast learning constraints.

[0012] Furthermore, the contrastive learning constraint between text and video The calculation formula is:

[0013] in, Indicates the similarity between calculated features, exp() indicates the exponential function, is the learnable temperature coefficient, represents the fused text features of video sample i in the batch training video, and Represents the global video features of two different video samples i and j.

[0014] Furthermore, the contrast learning constraint between the original video and the enhanced video The calculation formula is:

[0015] in, is the learnable temperature coefficient, and They represent the global features of the original video and the global features of the enhanced video of video sample i in the batch training video, Represents the global features of the enhanced video of video sample j in the batch training video.

[0016] Compared with the prior art, the technical solution of the present invention has the following beneficial effects: The present invention utilizes the multimodal capability of the multimodal large language model to generate supplementary text with rich details for each video, and then merges the supplementary text with the original text features in an additive manner, enriching the semantic information of the text modality expression, which not only reduces the dependence on manually annotated data, but also expands the semantic space of text expression. By generating enhanced videos for each video and using self-supervised contrastive learning methods, the expressive power of the video modality is further enhanced.

[0017] By constructing a "quadruple" consisting of original text, supplementary text, original video and enhanced video, and extracting video features based on the association between text and video through cross-attention encoding, we promote effective and meaningful information interaction and fusion between video and text, enhance the representation ability of the model, and enable the generated representation to more comprehensively and deeply reflect the inherent connection of multimodal data.

[0018] Two types of analogy learning constraints are designed: inter-modality contrast learning constraints and intra-modality contrast learning constraints. Inter-modality contrast learning constraints include original text-original video and original text-enhanced video contrast learning, which promotes effective alignment of modalities while taking into account the generalization ability of the model; intra-modality contrast learning uses self-supervised contrast learning of original video-enhanced video to force the model to learn the intrinsic structure and meaningful feature representation of the data, further improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the purpose and technical solution of the present invention clearer, the present invention provides the following drawings and descriptions: Figure 1 A flow chart of a method provided by an embodiment of the present invention; Figure 2 A schematic diagram of functional modules provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0021] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0022] The present invention provides a video-text retrieval method based on a multimodal large model, such as Figure 1 The figure shows a flow chart of a video-text retrieval method based on a multimodal large model. Figure 2 The figure shows a functional module schematic diagram of a video-text retrieval method based on a multimodal large model provided by the present invention, and the specific steps are as follows: S1: Get the original text-original video pair dataset; In a specific embodiment, an original text-original video pair dataset is obtained from the MSRVTT dataset, where the dataset contains 10,000 videos, and each video contains 20 manually annotated titles.

[0023] S2: Generate detailed supplementary text and enhanced video for the original video using a preset method; More specifically, the method for generating supplementary text in step S2 is to use a multimodal large language model to generate a detailed text description for the original video. The method for generating an enhanced video includes the following steps: sampling frames of the original video; applying image enhancement technology to the sampled video frames; recombining the processed frame sequence in order to generate an enhanced video, and the image enhancement technology includes at least one of the following: rotating, flipping, color transformation, spatial transformation, style transformation, and geometric transformation of the original video.

[0024] In a specific embodiment, samples are extracted from a data set as original text and original video. Taking one of the samples as an example, the original text corresponding to the original video is "People are singing on the beach". The original video is input into the multimodal large language model VideoLlava, and a text prompt is used: "Please generate a new caption for the video that not only rephrases the scene but also adds layers of interpretation or context that might not be explicit in the original caption. Think about elements that could be present in the scene in the video". The prompt guides the model to generate supplementary text with rich details: "As the sun sets on the beach, a group of people gather to sing ......The group is a mix of individuals, ......".

[0025] At the same time, the original video frame is rotated, flipped, and other operations are performed to obtain the enhanced video. Finally, the model input quadruple is constructed: original text, supplementary text, original video, and enhanced video.

[0026] S3: Calculate the fused text features of the original text and the supplementary text, and use cross-attention encoding to extract the video global features of the original video and the enhanced video respectively; More specifically, the specific method for calculating the fused text features in step S3 is: Use a word segmenter to segment the original text and the supplementary text into discrete words respectively; Mapping discrete words into continuous low-dimensional vectors through an embedding layer; Inputting the low-dimensional vector into a Transformer model to extract original text features and supplementary text features; The fused text features are obtained by summing the original text features and the supplementary text features.

[0027] The specific method of extracting the video global features in step S3 is: Sampling video frames from a video; Split each video frame into non-overlapping image blocks; Perform convolution operations on each image block to generate a low-dimensional vector and add position encoding and category labels; The low-dimensional vector with position encoding and category label is input into the video deep learning model, and the obtained category label is used as the feature vector of each video frame; The global features of the video are calculated by fusion text features and feature vectors of each video frame using cross attention encoding.

[0028] In a specific embodiment, for the original text and supplementary text First, we use a tokenizer to segment the text into a discrete set of words [“People”, “are”, “singing”, “on”, “the”, “beach”], then map the word set into a set of numbers [123, 456, 987, 565, 158, 681] through a predefined vocabulary, and then map the discrete information into a continuous low-dimensional vector through a feedforward neural network embedding layer. , where bs represents the batch size, which is set to 64, seq represents the sequence length, i.e. the number of tokens, which is set to 32, and dim represents the feature dimension, which is set to 512. This vector is only a vector representation of the text and does not contain feature information. After adding position encoding and category labels to the obtained vector, it is input into the Transformer model to extract the original text features. and supplementary text features , by summing up the text features, the fused text features are obtained .

[0029] For the original video or enhanced video, first sample video frames from the video and split each video frame into non-overlapping image blocks. In this embodiment, the resolution of the video is 224*224 and the size of the image block is 32*32, so each video frame will be split into 7*7 non-overlapping image blocks. These image blocks are converted into low-dimensional vectors through convolution operations, and position codes are added to each image block. In addition, a CLS token category label is added to each video frame. The position code is randomly initialized by a normal distribution at the beginning of training and remains fixed throughout the training process, while the CLStoken is randomly initialized by a normal distribution at each processing. Afterwards, these vectors are input into the VisionTransformer model, and finally the output of the CLS token is used as the feature vector of each video frame. .

[0030] Using cross-attention encoding, the text features are integrated As a request , the video frame features As value , Matrix transpose as key Through the cross attention mechanism, the similarity of each video frame is dynamically calculated and used as a dynamic weight. Based on the dynamic weight, the video frame features are fused to obtain the global features of the video. In this embodiment, 12 frames are sampled for each video and the text features are fused. By calculating the similarity with the 12 frame features, we get a set of 12 similarity weights , represents the similarity between the fused text feature and each video frame. Multiply the 12 similarities with the 12 frame features and add them up to get the global feature of the video. The calculation formula is:

[0031] in, Indicates Sample frames, For the The video frame features of sample frames, For the The similarity between the video frame features of the sampled frames and the fused text features can be calculated according to this method to calculate the global features of the original video. and enhance the global features of the video .

[0032] S4: Calculate contrastive learning constraints using the text features and the video global features; More specifically, the contrastive learning constraints in step S4 include cross-modal original text-original video contrastive learning constraints and original text-enhanced video contrastive learning constraints, as well as global feature constraints between original video and enhanced video within the modality.

[0033] Learning constraints for contrast between original text and original video across modalities , the calculation formula is:

[0034] Learning constraints for contrast between original text and augmented video across modalities , the calculation formula is:

[0035] Learning constraints for the contrast between original and enhanced videos within the modality , the calculation formula is:

[0036] in, Indicates the similarity between calculated features, exp() indicates the exponential function, and is the learnable temperature coefficient, represents the fused text features of video sample i in the batch training video, and Respectively represent the global features of video sample i and sample j in the batch training video, They represent the global features of the enhanced videos of video sample i and sample j in the batch training videos respectively.

[0037] S5: using the contrastive learning constraint to promote the model to achieve information alignment; In contrastive learning, the optimization goal is usually to minimize the constraints. Specifically, when using the three contrastive learning constraint formulas to promote the model to achieve information alignment, a key point is to balance the relationship between the numerator and the denominator. The denominator measures the dissimilarity between samples, that is, the similarity of negative sample pairs. Ideally, the smaller the denominator is expected to be, the higher the discrimination between negative sample pairs; while the numerator represents the similarity between the target sample and the positive sample pair. The larger the numerator is expected to be, the higher the match between the positive sample pairs. Therefore, the goal is to optimize the contrastive learning effect by adjusting the model so that the numerator increases and the denominator decreases.

[0038] S6: Generate feature information of the video to be retrieved using the trained model and save it. During retrieval, the similarity between the features of the input video or text and the saved feature information is calculated to obtain the retrieval result.

[0039] The present invention shows higher recall accuracy than other methods in existing video question-answering datasets such as MSRVTT and MSVD. The following experimental model code is implemented based on Pytorch and trained on a graphics card with 8 NVIDIA Tesla A100s, where the data batch size is 64, the maximum number of video sampling frames is 10, and the maximum text length is limited to 70. The weights of the pre-trained CLIP model ViT-B / 16 are used to initialize the weights of the text encoder and video encoder. In addition, the Adamax optimizer is used, and the learning rate is set to 4e-6. The experimental results on the MSRVTT and MSVD datasets are shown in Tables 1 and 2. Table 1 is the test results on the MSRVTT data, and Table 2 is the test results table on the MSVD data. In the table, CLIP2TV, Clip4Clip, X-CLIP, and CLIP-VIP are existing algorithm models. The additional data column indicates whether additional data needs to be added for training during the model training phase. R@1, R@5, and R@10 are recall rates, which indicate the accuracy of the search results in the top 1, 5, and 10 results. These indicators directly reflect the accuracy of the search system. The higher the value, the better the search effect. The MeanR column represents the average recall ranking, which measures the average position of the correct answer in the returned results. The smaller the value, the higher the correct answer. For data points that cannot be effectively measured, they are marked with '-'.

[0040] Table 1

[0041] Table 2

[0042] The present invention has demonstrated excellent performance in the field of video retrieval, especially in the two-way tasks of text retrieval of videos and video retrieval of texts. As can be seen from the tabular data, the present invention has achieved a very high level in multiple evaluation indicators such as R@1, R@5, and R@10, far exceeding other comparative models. At the same time, from the perspective of the average recall ranking indicator MeanR, the value of the present invention is lower, which means that the average ranking of the correct answers is higher, further verifying its excellent retrieval performance. It is worth mentioning that, without the use of additional manually annotated data, the performance of the present invention brings a 15.4% improvement in accuracy on the R@1 indicator of MSRVTT data compared to the CLIP-VIP model trained with an additional 10 million video-text pair data, proving that the use of a multimodal large language model to generate additional supplementary text effectively reflects video details and expands the expression space of the text. In addition, the results on the MSVD dataset show that compared with the Clip4Clip method that uses mean aggregation to obtain global video features, the use of the quadruple contrastive learning framework combined with cross-attention encoding brings a 22.1% R@1 accuracy improvement to the performance. This shows that extracting video features based on the association between text and video effectively promotes effective and meaningful information interaction and fusion between video and text, enhances the representation ability of the model, and enables the generated representation to more comprehensively and deeply reflect the inherent connection of multimodal data.

[0043] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A video-text retrieval method based on a multimodal large model, characterized in that: The steps include: Get the original text-original video pair dataset; Generate supplementary text and enhanced video for original video using preset methods; Calculate the fused text features of the original text and the supplemented text, and use cross-attention encoding to extract the global video features of the original video and the enhanced video respectively; Calculating contrastive learning constraints using the text features and the video global features; Using the contrastive learning constraint to promote the model to achieve information alignment; The trained model is used to generate feature information of the video to be retrieved and saved. During retrieval, the similarity between the features of the input video or text and the saved feature information is calculated to obtain the retrieval results.

2. A video-text retrieval method based on a multimodal large model according to claim 1, characterized in that: The method for generating supplementary text is to use a multimodal large language model to generate a detailed text description for the original video.

3. The video-text retrieval method based on a multimodal large model according to claim 1, characterized in that: The method for generating enhanced video comprises the following steps: sampling frames of original video; applying image enhancement technology to the sampled video frames; and recombining the processed frame sequences in order to generate enhanced video.

4. The video-text retrieval method based on a multimodal large model according to claim 3 is characterized in that: The image enhancement technology includes at least one of the following: rotation, flipping, color transformation, space transformation, style transformation, and geometric transformation of the original video.

5. The video-text retrieval method based on a multimodal large model according to claim 1, characterized in that: The specific method for calculating the fusion text features is: Use a word segmenter to segment the original text and the supplementary text into discrete words respectively; Mapping discrete words into continuous low-dimensional vectors through an embedding layer; Inputting the low-dimensional vector into a deep learning model to extract original text features and supplementary text features; The fused text features are obtained by summing the original text features and the supplementary text features.

6. The video-text retrieval method based on a multimodal large model according to claim 1, characterized in that: The specific method for extracting the global features of the video is: Sampling video frames from a video; Split each video frame into non-overlapping image blocks; Perform convolution operations on each image block to generate a low-dimensional vector and add position encoding and category labels; The low-dimensional vector with position encoding and category label is input into the video deep learning model, and the obtained category label is used as the feature vector of each video frame; The global features of the video are calculated by fusion text features and feature vectors of each video frame using cross attention encoding.

7. The video-text retrieval method based on a multimodal large model according to claim 1, characterized in that: The contrastive learning constraints include contrastive learning constraints between text and video across modalities, and global feature constraints between original video and enhanced video within a modality.

8. The video-text retrieval method based on a multimodal large model according to claim 7, characterized in that: The contrast learning constraints between text and video include original text-original video contrast learning constraints and original text-enhanced video contrast learning constraints.

9. The video-text retrieval method based on a multimodal large model according to claim 7, characterized in that: The contrastive learning constraints between text and video The calculation formula is: in, Indicates the similarity between calculated features, exp() indicates the exponential function, is the learnable temperature coefficient, represents the fused text features of video sample i in the batch training video, and Represents the global video features of two different video samples i and j.

10. The video-text retrieval method based on a multimodal large model according to claim 7, characterized in that: The contrastive learning constraint between the original video and the enhanced video The calculation formula is: in, is the learnable temperature coefficient, and They represent the global features of the original video and the global features of the enhanced video of video sample i in the batch training video, Represents the global features of the enhanced video of video sample j in the batch training video.

Images