Efficient video multi-scale text video retrieval method based on Mama

Abstract

The invention provides an efficient video multi-scale text video retrieval method based on Mama. The efficient video multi-scale text video retrieval method comprises the steps that S1, video multi-scale feature extraction is carried out; based on video features extracted by CLIP, by designing a convolution and pooling network, the features are mapped to different scales to complete multi-scale feature extraction; s2, video multi-scale feature fusion; performing cross-scale video feature fusion through scale fusion, time fusion and space fusion; and S3, efficient video multi-scale feature understanding based on the Mama architecture: for the extracted video long sequence, designing a residual Mama network based on the Mama architecture, and applying the extracted efficient video features to a text video retrieval task by using the linear calculation complexity of Mama. According to the method, the performance of the existing text-video retrieval method is further improved, and the method can be applied to the existing CLIP-based text-video retrieval method so as to enhance the retrieval capability of the existing CLIP-based text-video retrieval method.

Description

Technical Field

[0001] The present invention relates to the field of text-video retrieval, and in particular to an efficient video multi-scale text video retrieval method based on Mamba. Background Art

[0002] As the basis of multimodal tasks, the text-video retrieval task aims to align the feature spaces of text and video and establish a mapping relationship between the two to facilitate efficient and fast retrieval operations. With the continuous development of the Internet, this task is widely used in video search, video recommendation and other fields. In the context of the booming development of short videos today, this also means that this field is closely related to people's lives. Therefore, how to retrieve efficiently and accurately has always been an important problem to be solved in this field.

[0003] In recent years, with the continuous development of this field, migrating the large pre-trained model CLIP from image to text to video retrieval tasks has become the mainstream exploration direction, such as the CLIP4clip[1] method. However, the information contained in the video is far more than that in the image. It is difficult to obtain a rich video representation by simply applying CLIP to extract the image information of the video frame. In order to obtain accurate retrieval, mainstream methods capture cross-modal alignment at different granularities, including video-sentence (such as X-clip[2]), frame-sentence (such as X-pool[3]) and even tag-word (such as Wang et al. [4]). However, CLIP is essentially a flat structure with the same tag length between all layers. This also means that existing methods ignore the exploration of different scale representations of videos, which can provide more valuable contextual information for the comprehensive understanding of the video to enhance its retrieval effect.

[0004] The main problems and defects in the existing technology are: In general, although the existing methods have explored text-video retrieval methods that match at different granularities, they are limited by the flat structural properties of the CLIP features of their base pre-trained model and cannot fully understand and retrieve the video, which further limits the video retrieval performance.

[0005] The difficulty in solving the above problems and defects lies in: how to generate multi-scale video representation based on CLIP; and how to efficiently model the association of cross-scale video features.

[0006] The significance of solving the above problems and defects is: The multi-scale video feature extraction and understanding method further improves the performance of the existing text-video retrieval method, and proposes an efficient video feature cross-scale modeling method and proves that it has better computational overhead and performance than the Transformer architecture, making a feasible exploration of the substitutability of the Transformer architecture in text-video retrieval tasks. Summary of the invention

[0007] The present invention provides an efficient video multi-scale text-video retrieval method based on Mamba, which further improves the performance of existing text-video retrieval methods and can be used in existing CLIP-based text-video retrieval methods to enhance their retrieval capabilities.

[0008] The technical solution of the present invention is as follows: The Mamba-based efficient video multi-scale text video retrieval method of the present invention comprises the following steps: S1. video multi-scale feature extraction: based on the video features extracted by CLIP, the features are mapped to different scales by designing a convolution and pooling network to complete multi-scale feature extraction; S2. video multi-scale feature fusion: cross-scale video feature fusion is performed by scale fusion, time fusion and space fusion; S3. efficient video multi-scale feature understanding based on the Mamba architecture: for the extracted long video sequence, a residual Mamba network is designed based on the Mamba architecture, and the extracted efficient video features are used in the text video retrieval task by utilizing the linear computational complexity of Mamba.

[0009] Optionally, in the above-mentioned Mamba-based efficient video multi-scale text video retrieval method, it is characterized in that in step S1, for the visual features extracted by CLIP, in order to obtain fine-grained video information, all visual markers are used as video-level representations, that is, video features ,in is the frame number, Indicates the number of visual markers; based on visual features , extract multi-scale video features ,in Indicates Frame at scale The following feature representation: s ∈ [1, S ] ; for features Perform convolution or pooling operations to obtain , as follows: .

[0010] Optionally, in the above-mentioned Mamba-based efficient video multi-scale text video retrieval method, in step S2, in scale fusion, firstly, video features from the same scale are fused in time order, and then video tags of the same scale are rearranged into a one-dimensional sequence; finally, these tags are connected in order from low resolution to high resolution, and after fusion, the video features are expressed as: ; In temporal fusion, the tags from the same video frame are first fused and then rearranged into a one-dimensional sequence in scale order; finally, these video tags are concatenated in temporal order, and the fused video features are expressed as: ; In spatial fusion, pooling is first performed through the temporal dimension to fuse the temporal information and retain only the spatial information; then these markers are rearranged in a temporal fusion manner, and the final video feature representation is: .

[0011] Optionally, in the above-mentioned Mamba-based efficient video multi-scale text video retrieval method, in step S3, a residual network is designed to modify Mamba into an effective multi-scale video learner, and the residual network is expressed as: , A single linear layer initialized to zero is used as the gating function after the Mamba module. , the learning process is expressed as: in, Indicated in The first of the Mamba layers Layer, for Mamba modules, is the evolution parameter, and are the projection parameters.

[0012] According to the technical solution of the present invention, the beneficial effects produced are: The method of the present invention designs an efficient video multi-scale feature extraction and understanding framework based on the Mamba architecture to provide more abundant and accurate video features for the existing text-video retrieval model to enhance its retrieval effect.

[0013] In order to better understand and illustrate the concept, working principle and effect of the present invention, the present invention is described in detail below through specific embodiments in conjunction with the accompanying drawings: BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0015] Figure 1 It is a flowchart of the steps of the Mamba-based efficient video multi-scale text video retrieval method of the present invention; Figure 2 It is a network architecture diagram involved in the Mamba-based efficient video multi-scale text video retrieval method of the present invention; Figure 3 This is a comparison chart of the performance and efficiency of different architectures; Figure 4 This is a comparison chart of the video memory of Transformer and Mamba at different frame rates. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical method and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. These examples are only illustrative and not limiting of the present invention.

[0017] The invention discloses an efficient video multi-scale text-video retrieval method based on Mamba, and provides an efficient video multi-scale feature extraction and understanding framework (abbreviated as MUSE) based on Mamba architecture. This is a multi-scale Mamba framework with linear computational complexity. During the training process, it can be applied as a plug-and-play module to the existing text-video retrieval method, and the retrieval performance of the existing text-video retrieval method is enhanced through the video multi-scale features.

[0018] Without loss of generality, for the text-video retrieval task, assuming that a video and the corresponding query , the existing method uses CLIP to extract image features for each frame. For the text branch, the [EOT] tag is selected as the text representation. For the visual branch, the [CLS] tag of each frame is regarded as the video level representation . Finally, the optimization objective of the model can be expressed as:

[0019] in, Represents an unmatched text query.

[0020] The Mamba-based efficient video multi-scale text video retrieval method of the present invention comprises the following steps: S1. Video multi-scale feature extraction. Based on the video features extracted by CLIP, the convolution and pooling networks are designed to map the features to different scales to complete multi-scale feature extraction. In the extraction of multi-scale features of videos, we focus on exploring the impact of the combination of features of different scales on model performance and training overhead, with the aim of reducing the computational overhead of model training while maintaining performance.

[0021] Specifically, for the visual features extracted by CLIP, in order to obtain fine-grained video information, the present invention uses all visual markers as video-level representations, namely, video features ,in is the frame number, represents the number of visual markers. On this basis, Figure 2 As shown, based on visual features , the method of the present invention extracts multi-scale video features ,in Indicates Frame at scale The following feature representation: s ∈ [1, S ] In order to get The method of the present invention has the following characteristics: Perform convolution or pooling operations as follows:

[0022] S2. Video multi-scale feature fusion: Cross-scale video feature fusion is performed through scale fusion, temporal fusion and spatial fusion.

[0023] Based on the extracted multi-scale video features, we explore different feature arrangement methods to obtain a comprehensive video feature representation. In the arrangement of multi-scale video features, we focus on different arrangement methods and scanning methods when inputting the Mamba architecture, and prioritize scanning video features of different scales according to feature scale, time, and space dimensions to obtain the optimal multi-scale representation. Among them:

[0024] Scale fusion: First, video features from the same scale are fused in temporal order, and then video tags of the same scale are rearranged into a one-dimensional sequence, such as Figure 2 As shown in (a) in . Finally, these markers are connected in order from low resolution to high resolution. After fusion, the video features can be expressed as:

[0025] Temporal fusion: Different from the scale fusion method, it first fuses the tags from the same video frame and then rearranges them into a one-dimensional sequence according to the scale order, such as Figure 2 As shown in (b); Finally, these video tags are connected in time order. In this way, the fused video features can be expressed as: Spatial fusion: The fusion order of this method is the same as that of temporal fusion on each frame. The difference is that pooling is first performed through the temporal dimension to fuse the temporal information and keep only the spatial information. Then, these markers are rearranged in a temporal fusion manner. This method focuses on the spatial dimension, so it is called spatial fusion. Figure 2 As shown in (c) in the figure, the final video feature can be expressed as:

[0026] In practical applications, it is found that rearranging multi-scale video features into a one-dimensional sequence in a scale fusion manner can achieve the best performance and is used as the final implementation of the method.

[0027] S3. Efficient video multi-scale feature understanding based on Mamba architecture. For the extracted video sequence, the residual Mamba network is designed based on the Mamba architecture. By utilizing the linear computational complexity of Mamba (the traditional Transformer self-attention mechanism has a quadratic complexity), efficient video representation extraction is achieved, and the extracted efficient video features are used in the text-video retrieval task. In the understanding of multi-scale video features, the efficiency of feature modeling is emphasized. That is, different architectures are compared in terms of computational complexity, inference speed, and computing resource usage.

[0028] In order to modify the Mamba module into an effective multi-scale video learner, a residual network is designed, which can be expressed as: Through experiments, it is found that the gated residual architecture works best in multi-scale video learning. In actual design, the present invention uses a single linear layer initialized to zero as the gating function after the Mamba module. , the learning process can be expressed as:

[0029] in, Indicated in The first of the Mamba layers For the Mamba module, is the evolution parameter, and are the projection parameters.

[0030] For the application of Mamba architecture in text-video retrieval tasks, we focus on exploring the performance gain and computational overhead brought by different network structure designs to find the balance between performance and efficiency. To verify the effectiveness of the method, we selected several mainstream text-video retrieval methods and added the MUSE network proposed in this invention to them for training, and all of them achieved considerable performance improvement.

[0031] Experimental settings: For the dataset MSR-VTT, the proposed method sets the input video frame length to 12 and the text length to 32. For model fine-tuning, the proposed method keeps the training hyperparameters and settings of the base model unchanged and trains the MUSE network with a learning rate of 10 times (for example, the original learning rate of CLIP4clip is 1e-4, and the learning rate of MUSE is 1e-3). The number of layers of the ResMamba module is set to 4, and the scale features are selected as {1, 3, 7, 14}. All experiments are performed on 8 NVIDIA A100 GPUs.

[0032] Evaluation indicators: To evaluate the method, the proposed method uses standard retrieval indicators to test performance, including top K recall (R@K, the higher the better), median recall (MdR, the lower the better), and average recall (MnR, the lower the better). R@K represents the recall percentage of samples that find the correct answer in the top K search results. The proposed method sets K to {1,5, 10}. MdR is defined as the median ranking of the true results in the result ranking list, while MnR is defined as the average ranking of all correct results.

[0033] Experimental results: (1) Effectiveness: To verify the effectiveness of the proposed method, four mainstream text-video retrieval methods (CLIP4clip, EMCL-Net, STAN, T-MASS) were tested on the MSR-VTT dataset, and the MUSE network was added to the training process to compare their performance. As shown in Table 1, for different methods, after adding the MUSE network proposed in the present invention, the retrieval performance of text-video and video-text was significantly improved, which also verified the generalization and effectiveness of the proposed method.

[0034] Table 1. Performance improvement of MUSE network on various text-video retrieval methods

[0035] (2) Efficiency: To verify the efficiency of the method of the present invention, the Mamba architecture used in the method of the present invention is compared with a variety of mainstream architectures, such as Transformer, Flash Attention, and MambaOut. Figure 3 As shown in the figure, under the same video multi-scale understanding modeling conditions, the Mamba architecture used by the method of the present invention has the best performance among the four architectures, and also has good performance in terms of parameter quantity and video memory occupancy. In general, the Mamba architecture has a more efficient multi-scale video modeling and understanding capability. In addition, the method of the present invention also makes a detailed comparison of the video memory occupancy when different video frame numbers are used as input. Figure 4 As shown in the figure, while expanding the number of video frame inputs, the video memory overhead brought by the Transformer is in a quadratic relationship with the number of frames, so it will soon exceed the original video memory limit. Due to its linear complexity, Mamba can save 79.9% of the video memory overhead compared to the Transformer when the input is 16 frames. Figure 3 and Figure 4 Together, the effectiveness of the method and architecture design proposed in this invention is demonstrated.

[0036] The above description is the best embodiment based on the concept and working principle of the invention. The above embodiment should not be understood as limiting the protection scope of the present claims, and other implementations and combinations of implementations according to the concept of the present invention belong to the protection scope of the present invention.

[0037] References [1] Luo H, Ji L, Zhong M, et al. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning[J]. Neurocomputing, 2022, 508: 293-304. [2] Ma Y, Xu G, Sun X, et al. X-clip: End-to-end multi-grainedcontrastive learning for video-text retrieval[C] / / Proceedings of the 30th ACM International Conference on Multimedia. 2022: 638-647. [3] Gorti S K, Vouitsis N, Ma J, et al. X-pool: Cross-modal language-video attention for text-video retrieval[C] / / Proceedings of the IEEE / CVFconference on computer vision and pattern recognition. 2022: 5006-5015. [4] Wang Z, Sung Y L, Cheng F, et al. Unified coarse-to-finealignment for video-text retrieval[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2023: 2816-2827.

Claims

1. An efficient video multi-scale text retrieval method based on Mamba, characterized in that: The following steps are involved: S1. Video multi-scale feature extraction: Based on the video features extracted by CLIP, the features are mapped to different scales by designing convolution and pooling networks to complete multi-scale feature extraction; S2. Video multi-scale feature fusion; cross-scale video feature fusion through scale fusion, time fusion and space fusion; S3. Efficient video multi-scale feature understanding based on Mamba architecture: For the extracted long video sequences, a residual Mamba network is designed based on the Mamba architecture. By utilizing the linear computational complexity of Mamba, the extracted efficient video features are used in text video retrieval tasks.

2. The Mamba-based efficient video multi-scale text retrieval method according to claim 1, characterized in that: In step S1, for the visual features extracted by CLIP, in order to obtain fine-grained video information, all visual markers are used as video-level representations, namely, video features ,in is the frame number, Indicates the number of visual markers; based on visual features , extract multi-scale video features ,in Indicates Frame at scale The following feature representation: ; Features Perform convolution or pooling operations to obtain , as follows: 。 3. The Mamba-based efficient video multi-scale text video retrieval method according to claim 2, characterized in that: In step S2, in the scale fusion, firstly, the video features from the same scale are fused in time order, and then the video tags of the same scale are rearranged into a one-dimensional sequence; finally, these tags are connected in order from low resolution to high resolution. After fusion, the video features can be expressed as: ； In the temporal fusion, the tags from the same video frame are first fused, and then they are rearranged into a one-dimensional sequence in scale order; finally, these video tags are connected in time order, and the fused video features are expressed as: ； In the spatial fusion, pooling is first performed through the time dimension to fuse the temporal information and retain only the spatial information; then the markers are rearranged in a temporal fusion manner, and the final video feature is represented as: 。 4. The Mamba-based efficient video multi-scale text video retrieval method according to claim 2, characterized in that: In step S3, a residual network is designed to modify the Mamba into an effective multi-scale video learner. The residual network is expressed as: ， A single linear layer initialized to zero is used as the gating function after the Mamba module. , the learning process is expressed as: in, Indicated in The first of the Mamba layers Layer, for Mamba modules, is the evolution parameter, and are the projection parameters.

Images