Video-text retrieval method based on global and local semantics
Through the combination of global and local semantic interaction modules and loss functions, the existing video-text retrieval methods have solved the problem of high computational cost and low efficiency, and achieved efficient video-text retrieval performance improvement.
Patent Information
- Application Number
- CN202411020426.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
AI Technical Summary
Existing video-text retrieval methods are cost-effective, inefficient in processing complex time series data, and are difficult to effectively capture dynamic information in video, resulting in unsatisfactory results.
Using the video-text search method based on global and local semantics, a video-text cross-modal matching model is constructed, and the global and local semantic information of video-text is captured through the global interaction module and the local interaction module, and the final generation is generated through the similarity fusion module. The degree of matching, introduces mutual consistency loss and internal diversity loss to improve semantic alignment and distinction.
It significantly improves the performance of video-text retrieval, reduces computational costs, verifies the effectiveness and superiority of the method, can efficiently capture potential shared semantics across modalities, and improves retrieval efficiency and accuracy.
Smart Images

Figure CN118916520A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of cross-modal retrieval, and more specifically, relates to a video-text retrieval method based on global and local semantics. Background Art
[0002] In recent years, cross-modal learning has made significant progress in vision and language tasks, especially in the field of text-image retrieval. Large-scale image-text pre-training models (such as CLIP, Contrastive Language-lmage Pre-Training) have demonstrated strong performance in different tasks. These models achieve semantic alignment between different modalities by converting images and texts into a common embedding space. However, text-video retrieval is more challenging than text-image retrieval because videos contain richer temporal and spatial information. Existing methods mainly map videos and texts into a common embedding space and use cross-modal interactions of specific entities for semantic alignment.
[0003] Although existing methods have achieved many outstanding research results, these methods are usually computationally expensive, inefficient, and may not work well when processing complex time series data. Compared with image data, video data has higher dimensions and more complex internal structures, which makes it difficult to directly apply simple image-text methods to video data. Some current methods try to capture dynamic information in videos by introducing attention mechanisms or more complex network structures, but these methods often require a lot of computing resources and time. Summary of the invention
[0004] The purpose of the present invention is to overcome the shortcomings of the prior art, provide a video-text retrieval method based on global and local semantics, construct a video-text cross-modal matching model, and optimize video-text retrieval by mining cross-modal potential shared semantics.
[0005] In order to achieve the above-mentioned object of the invention, the video-text retrieval method based on global and local semantics of the present invention comprises the following steps:
[0006] S1: Collect several video-text training sample pairs according to actual needs to form a training sample set, where each training sample pair includes a text X and a video V corresponding to the text;
[0007] S2: Construct a video-text cross-modal matching model, including a text feature extraction module, a video feature extraction module, a global interaction module, a local interaction module and a similarity fusion module, where:
[0008] The text feature extraction module is used to normalize the sequence length of text X to the preset text length NW , and then use the text encoder to W Encode words to get word features D represents the preset encoding feature dimension, and uses the sequence end mark feature As the global feature of text X, then the global feature F X Send it to the global interaction module and convert the word feature F W Send to the local interaction module;
[0009] The video feature extraction module is used to sample the video frames in the video V to obtain N f Frame video frame, and then use the image encoder to f Encode the video frame by frame to get the video features And sent to the global interaction module and the local interaction module;
[0010] The global interaction module is used to calculate the video-text global semantic similarity Sim1 and send it to the similarity fusion module. The specific method is as follows:
[0011] The softmax function is used to calculate the weight α between text features and frame features. k :
[0012]
[0013] Where τ represents the temperature adjustment coefficient, and the superscript T represents the transposition;
[0014] Based on the weight α k Aggregate frame features to get text-guided video features
[0015]
[0016] Calculate text feature F X and video features The similarity between them is taken as the video-text global semantic similarity Sim1;
[0017] The local interaction module is used to calculate the video-text local semantic similarity Sim2 and send it to the similarity fusion module; the local interaction module includes a shared Transformer encoder and a similarity calculation module, where:
[0018] The shared Transformer encoder includes N L The stacked Transformer encoding blocks are used to encode the word features F W and video feature F V Perform feature encoding to obtain fine-grained conceptual semantic features C of text and video X and C V, the feature encoding methods are:
[0019] When extracting the semantic features of text concepts, word features F are used W As keys and values, use the text to be learned parameter matrix As a query, N q Represents the preset dimension, and uses the query output by the last layer as the text concept semantic feature And sent to the similarity calculation module.
[0020] When extracting the semantic features of video concepts, the video feature F is used V As keys and values, the video parameter matrix to be learned is used As a query, the query output by the last layer is used as the video concept semantic feature And send it to the similarity calculation module;
[0021] The similarity calculation module is used to calculate the text concept semantic features C X and video concept semantic features C V The similarity is taken as the video-text local semantic similarity Sim2;
[0022] The similarity fusion module is used to fuse the video-text global semantic similarity Sim1 and the video-text local semantic similarity Sim2 to obtain the final matching degree S;
[0023] S3: using the training sample set in step S1 to train the video-text cross-modal matching model to obtain a trained video-text cross-modal matching model;
[0024] S4: When video-text retrieval is required, the text / video to be retrieved and the alternative videos / texts are input into the video-text cross-modal matching model trained in step S3 in pairs to obtain the corresponding matching degree, and then the video / text matching the text / video to be retrieved is screened out based on the matching degree.
[0025] The present invention is based on a video-text retrieval method of global and local semantics, and constructs a video-text cross-modal matching model comprising a text feature extraction module, a video feature extraction module, a global interaction module, a local interaction module and a similarity fusion module. The global interaction module is used to calculate the global similarity of text features and video features extracted by the text feature extraction module and the video feature extraction module, the local interaction module is used to calculate the local similarity of word features and video features of the text based on a Transformer encoder, and then the similarity fusion module fuses the global similarity and the local similarity to obtain the video-text matching degree.
[0026] The present invention has the following beneficial effects:
[0027] 1) The present invention designs a parameter-free global interaction module and a shared local interaction module. These two modules can efficiently capture global and local semantic information and optimize text-video retrieval by mining cross-modal potential shared semantics.
[0028] 2) The present invention also proposes mutual consistency loss and internal diversity loss to further improve semantic alignment and distinguishability;
[0029] 3) Through experiments, it is found that the present invention not only achieves significant improvement in performance, but also greatly reduces the computational cost, verifying its effectiveness and superiority. BRIEF DESCRIPTION OF THE DRAWINGS
[0030] Figure 1 It is a flowchart of a specific implementation method of the video-text retrieval method based on global and local semantics of the present invention;
[0031] Figure 2 is a structural diagram of the video-text matching model in the present invention;
[0032] Figure 3 It is a comparison chart of the number of parameters of the present invention and the comparative method in Example 1. DETAILED DESCRIPTION
[0033] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.
[0034] Example
[0035] Figure 1 FIG. 1 is a flowchart of a specific implementation of the video-text retrieval method based on global and local semantics of the present invention. Figure 1 As shown, the specific steps of the video-text retrieval method based on global and local semantics of the present invention include:
[0036] S101: Obtain video-text training sample pairs:
[0037] Video-text training sample pairs are collected according to actual needs, where each training sample pair includes a text X and a video V corresponding to the text.
[0038] S102: Constructing a video-text cross-modal matching model:
[0039] In the present invention, a video-text matching model is constructed based on global and local semantic consistency. Figure 2 It is a structural diagram of the video-text matching model in the present invention. Figure 2As shown, the video-text cross-modal matching model in the present invention includes a text feature extraction module, a video feature extraction module, a global interaction module, a local interaction module and a similarity fusion module. Each module is described in detail below.
[0040] The text feature extraction module is used to normalize the sequence length of text X to the preset text length N W , and then use the text encoder to W Encode the words to get word features D represents the preset encoding feature dimension, and uses the end of sequence feature As the global feature of text X, then the global feature F X Send it to the global interaction module and convert the word feature F W The text encoder is sent to the local interaction module. The specific structure of the text encoder can be set according to actual needs. In this embodiment, the text encoder uses the text encoder in the pre-trained CLIP model.
[0041] The CLIP model is a multimodal pre-trained neural network released by OpenAl in 2021. It is an effective and scalable method for learning from natural language supervision. The core idea of the model is to use a large amount of paired data of images and texts for pre-training to learn the alignment relationship between images and texts. The CLIP model consists of two main parts: 1. Text Encoder: used to convert text into a high-dimensional vector representation. 2. Image Encoder: used to convert images into similar vector representations. The specific principles and working process of the CLIP model can be found in the literature "Radford A, Kim JW, Hallacy C, et al. Learning Transferable Visual Models From Natural Language Supervision[J]. 2021. DOI: 10.48550 / arXiv.2103.00020."
[0042] The video feature extraction module is used to sample the video frames in the video V to obtain N f Frame video frame, and then use the image encoder to f Encode the video frame by frame to get the video features And sent to the global interaction module and the local interaction module. The specific method of the video feature extraction module can be set according to actual needs. In this embodiment, the image encoder uses the image encoder in the pre-trained CLIP model.
[0043] The Global Interaction Module (GIM) is used to capture coarse-grained semantic information between text and video data, achieve global semantic alignment at the sentence-video level, obtain the global semantic similarity Sim1 of text features and video features and send it to the similarity fusion module. The specific method is:
[0044] The core idea of the global interaction model in this invention is to use text features to guide the weighted aggregation of video frame features, so as to obtain text-guided video feature representation. That is, the weight α between text features and frame features is calculated using the softmax function. k :
[0045]
[0046] Among them, τ represents the temperature adjustment coefficient, which is used to adjust the smoothness of the softmax function, and the superscript T represents the transpose.
[0047] Based on the weight α k Aggregate frame features to get text-guided video features
[0048]
[0049] Calculate text feature F X and video features The similarity between them is taken as the video-text global semantic similarity Sim1. In this embodiment, cosine similarity is used, and the calculation formula of the video-text global semantic similarity Sim1 is:
[0050]
[0051] The Local Interaction Module (LIM) is used as a fine-grained interaction mechanism to encourage the conceptual semantic sharing between text and video data, thereby obtaining the video-text local semantic similarity Sim2 and sending it to the similarity fusion module. Figure 2 As shown, the local interaction module in the present invention includes a shared Transformer encoding block and a similarity calculation module. Next, each module is described in detail.
[0052] The shared Transformer encoder includes N L The stacked Transformer encoding blocks are used to encode the word features F W and video feature F V Perform feature encoding to obtain fine-grained conceptual semantic features C of text and video X and CV , the feature encoding methods are:
[0053] When extracting the semantic features of text concepts, word features F are used W As key and value, use the text parameter matrix to be learned As a query, N q Represents the preset dimension, and uses the query output by the last layer as the text concept semantic feature And sent to the similarity calculation module.
[0054] When extracting the semantic features of video concepts, the video feature F is used V As keys and values, the video parameter matrix to be learned is used As a query, the query output by the last layer is used as the video concept semantic feature And sent to the similarity calculation module.
[0055] The Transformer encoder is a commonly used encoder, and its principle and processing process will not be described here.
[0056] The similarity calculation module is used to calculate the text concept semantic features C X and video concept semantic features C V The similarity is taken as the video-text local semantic similarity Sim2. In this embodiment, the calculation formula of the video-text local semantic similarity Sim2 is:
[0057]
[0058] The similarity fusion module is used to fuse the video-text global semantic similarity Sim1 and the video-text local semantic similarity Sim2 to obtain the final matching degree S. The specific formula for similarity fusion in this embodiment is:
[0059] S=Sim1+δSim2
[0060] Wherein, δ represents the preset weight.
[0061] S103: Training video-text cross-modal matching model:
[0062] The video-text cross-modal matching model is trained using the training sample set in step S1 to obtain a trained video-text cross-modal matching model.
[0063] The calculation of the loss function is very important for model training. Although the present invention shares the Transformer encoder for text and video in the local interaction module, there may still be a "semantic asymmetry" problem between the generated visual and text concepts. To solve this problem, the mutual consistency loss (ICL) is proposed in this embodiment to encourage each text concept in the text encoding feature in the local interaction module to be close to the corresponding video concept in its video encoding feature, thereby promoting better semantic alignment during model training. The calculation method of the mutual consistency loss is:
[0064] The following formula is used to calculate the distance between the text concept semantic feature and the corresponding concept in the video concept semantic feature: b [i]:
[0065] dis b [i]=||C b,X [i]-C b,V [i]||2
[0066] Among them, C b,X , C b,V represents the text concept semantic features and video concept semantic features of the b-th training sample pair in the current training batch, b = 1, 2, ..., B, B represents the number of training sample pairs in a single training batch, C b,α [i] represents the concept semantic feature C b,α The i-th row vector in , α=X,V, || ||2 means to find the bi-norm.
[0067] Then, to avoid overfitting between the two types of concepts, a semantic regularization γ is designed by constraining the positive video-text concepts within an appropriate range. b :
[0068]
[0069] Among them, λ represents the relaxation factor, which is used to control the minimum semantic distance between text-video concept pairs, ensuring that these concept pairs maintain a certain degree of semantic distinction so as to better distinguish different concepts.
[0070] Mutual consistency loss L ICL The calculation formula is:
[0071]
[0072] In addition, in order to alleviate the overlapping clustering of matching video-text concepts learned based on mutual consistency loss, it is further proposed to promote the internal differentiation of visual and text concepts to effectively solve the problem of multi-context collapse. In this embodiment, an internal diversity loss (IDL) is also introduced. The specific calculation method is:
[0073] For the concept semantic feature C b,α , calculate its different concepts C b,α [i] C b,α The correlation between [i′] b,α (i,i′), i,i′=1,2,…,N q In this embodiment, the correlation adopts cosine similarity, that is:
[0074] con b,α (i,i′)=cos(C b,α [i],C b,α [i′])
[0075] Then a triplet-like loss function is used to enlarge the distance between these concepts, so the internal diversity loss L IDL The calculation formula is:
[0076]
[0077] This embodiment also uses the conventional contrastive learning loss I CL , and its calculation formula is:
[0078]
[0079] Among them, S(X b ,V b ) represents the text X in the b-th training sample pair b and video V b The matching degree between them, e represents the natural constant, π represents the temperature coefficient, which is used to adjust the distribution of the matching degree.
[0080] The calculation formula of the loss function LOSS finally used in this embodiment is:
[0081] LOSS=w1I CL +w2L ICL +w3L IDL
[0082] Among them, w1, w2, and w3 represent preset weights respectively.
[0083] S104: Video-text retrieval:
[0084] When video-text retrieval is required, the text / video to be retrieved and the alternative videos / texts are input into the video-text cross-modal matching model trained in step S3 in pairs to obtain the corresponding matching degree, and then the video / text matching the text / video to be retrieved is screened out based on the matching degree.
[0085] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.
[0086] Example 1
[0087] In this embodiment, the experimental conditions are set as follows: system: Ubuntu 20.04, software: Python 3.6, processor: Intel(R) Xeon(R) Gold 5220R CPU@2.20GHz, memory: 512GB.
[0088] In this embodiment, eight existing video-text methods are selected as comparison methods, including CLIP4Clip, CenterCLIP, X-CLIP, TS2-Net, X-Pool, CLIP-VIP, DiCoSA and Prompt Switch. Figure 3 is a comparison chart of the number of parameters of the present invention and the comparative method in Example 1. Although the existing methods have achieved significant performance improvements, they have encountered high computational costs, resulting in low retrieval efficiency in practical applications.
[0089] Example 2
[0090] In this embodiment, the experimental conditions are set as follows: system: Ubuntu 20.04, software: Python 3.6, processor: Intel(R) Xeon(R) Gold 5220R CPU@2.20GHz, memory: 512GB.
[0091] Table 1 is a comparison table of performance statistics of the present invention and the comparative method on the MSR-VTT data set in this embodiment.
[0092]
[0093]
[0094] Table 1
[0095] As shown in Table 1, Table 1 shows the retrieval performance and computational cost (number of floating point operations) of the present invention and the comparative method on CLIP-ViT-B / 32 and CLIP-ViT-B / 16. The * indicates that QBNORM (Querybank Normalization) is used. In general, the present invention method achieves the best balance between performance and computational cost. In particular, based on CLIP-ViT-B / 32 without QBNORM, the present invention performs best among all models, with an R@1 of 48.1% for text-to-video retrieval and 47.1% for video-to-text retrieval. In addition, compared with the most advanced Prompt Switch method, the present invention method has lower computational cost, which is reduced by about 220 times. Moreover, at approximately the same computational cost, the present invention improves DiCoSA by 1.4% and 3.0% in R@1 for text-to-video and video-to-text retrieval, respectively. After using QBNORM, the present invention method further improves the performance. After being equipped with a more powerful visual backend CLIP-ViT-B / 16, the performance of the proposed method is further improved and the performance is comparable to that of other methods. The above results clearly demonstrate the effectiveness of the proposed method.
[0096] In order to demonstrate the robustness of the present invention, this embodiment further verifies the generalization performance of the present invention on more data sets, and experiments are conducted on the MSVD, DiDeMo, LSMDC and ActivityNet data sets. Table 2 is a comparison table of the performance statistics of the present invention and the comparative method on different data sets in this embodiment.
[0097]
[0098]
[0099] Table 2
[0100] Table 2 shows the performance comparison results of text-to-video retrieval and video-to-text retrieval. All models only use CLIP-ViT-B / 32 and no post-processing operations are performed. It can be observed from Table 2 that the present invention continues to improve the performance on the MSVD and DiDeMo datasets, and achieves comparable results on the LSMDC and ActivityNet datasets. For example, on the MSVD and ActivityNet datasets, the present invention exceeds the best comparison method Prompt Switch in R@1 (e.g., 47.5% vs. 47.1%, 23.4% vs. 23.1 on t2v; 69.1% vs. 68.5%, 22.2% vs. 22.0 on v2t). For the DiDeMo dataset, the present invention also achieves the best results compared with existing methods. In addition, some failure cases are also found. For example, on the ActivityNet dataset, the present invention fails to bring significant performance improvements in both t2v and v2t directions. The possible reason is that ActivityNet involves more complex and longer movie videos, and the optimal hyperparameters of the present invention on this dataset may be different, while the present invention fixes them on all datasets and baselines (e.g., the number of learnable queries). Due to limited space, the computational costs of these methods are omitted in this table, and the approximate number of floating-point operations is comparable to the values reported in Table 1, indicating that the present invention is a more efficient approach.
[0101] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.
Claims
1. A video-text retrieval method based on global and local semantics, characterized in that: The following steps are involved: S1: Collect several video-text training sample pairs according to actual needs to form a training sample set, where each training sample pair includes a text X and a video V corresponding to the text; S2: Construct a video-text cross-modal matching model, including a text feature extraction module, a video feature extraction module, a global interaction module, a local interaction module, and a similarity fusion module. in: The text feature extraction module is used to normalize the sequence length of text X to the preset text length N W , and then use the text encoder to W Encode words to get word features D represents the preset encoding feature dimension, and uses the sequence end mark feature As the global feature of text X, then the global feature F X Send it to the global interaction module and convert the word feature F W Send to the local interaction module; The video feature extraction module is used to sample the video frames in the video V to obtain N f Frame video frame, and then use the image encoder to f Encode the video frame by frame to get the video features And sent to the global interaction module and the local interaction module; The global interaction module is used to calculate the video-text global semantic similarity Sim1 and send it to the similarity fusion module. The specific method is as follows: The softmax function is used to calculate the weight α between text features and frame features. k : Where τ represents the temperature adjustment coefficient, and the superscript T represents the transposition; Based on the weight α k Aggregate frame features to get text-guided video features Calculate text feature F X and video features The similarity between them is taken as the video-text global semantic similarity Sim1; The local interaction module is used to calculate the video-text local semantic similarity Sim2 and send it to the similarity fusion module; the local interaction module includes a shared Transformer encoder and a similarity calculation module, where: The shared Transformer encoder includes N L The stacked Transformer encoding blocks are used to encode the word features F W and video feature F V Perform feature encoding to obtain fine-grained conceptual semantic features C of text and video X and C V , the feature encoding methods are: When extracting the semantic features of text concepts, word features F are used W As keys and values, use the text to be learned parameter matrix As a query, N q Represents the preset dimension, and uses the query output by the last layer as the text concept semantic feature And sent to the similarity calculation module. When extracting the semantic features of video concepts, the video feature F is used V As keys and values, the video parameter matrix to be learned is used As a query, the query output by the last layer is used as the video concept semantic feature And send it to the similarity calculation module; The similarity calculation module is used to calculate the text concept semantic features C X and video concept semantic features C V The similarity is taken as the video-text local semantic similarity Sim2; The similarity fusion module is used to fuse the video-text global semantic similarity Sim1 and the video-text local semantic similarity Sim2 to obtain the final matching degree S; S3: using the training sample set in step S1 to train the video-text cross-modal matching model to obtain a trained video-text cross-modal matching model; S4: When video-text retrieval is required, the text / video to be retrieved and the alternative videos / texts are input into the video-text cross-modal matching model trained in step S3 in pairs to obtain the corresponding matching degree, and then the video / text matching the text / video to be retrieved is screened out based on the matching degree.
2. The video-text retrieval method according to claim 1, characterized in that: The text encoder in step S2 uses the text encoder in the pre-trained CLIP model.
3. The video-text retrieval method according to claim 1, characterized in that: In step S2, the video encoder uses the image encoder in the pre-trained CLIP model.
4. The video-text retrieval method according to claim 1, characterized in that: In step S2, the feature similarity adopts cosine similarity.
5. The video-text retrieval method according to claim 1, characterized in that: The calculation formula for similarity fusion in step S2 is: S=Sim1+δSim2 Wherein, δ represents the preset weight.
6. The video-text retrieval method according to claim 1, characterized in that: The calculation formula of the loss function LOSS in the video-text cross-modal matching model training process in step S3 is: LOSS=w1I CL +w2L ICL +w3L IDL Among them, w1, w2, and w3 represent the preset weights, I CL represents the contrastive learning loss, which is calculated as: b=1,2,…,B, B represents the number of training sample pairs in a single training batch, S(X b ,V b ) represents the text X in the b-th training sample pair b and video V b The matching degree between them, e represents the natural constant, and π represents the temperature coefficient; L ICL represents the mutual consistency loss, which is calculated as: The following formula is used to calculate the distance between the text concept semantic feature and the corresponding concept in the video concept semantic feature: b [i]: dis b [i]=||C b,X [i]-C b,V [i]||2 Among them, C b,X , C b,V represents the text concept semantic features and video concept semantic features of the b-th training sample pair in the current training batch, C b,α [i] represents the concept semantic feature C b,α The i-th row vector in , α=X,V, || ||2 means to find the bi-norm; Compute semantic regularization γ b : Where λ represents the relaxation factor; Then the mutual consistency loss L is calculated using the following formula ICL : L IDL represents the internal diversity loss, which is calculated as: For the concept semantic feature C b,α , calculate its different concepts C b,α [i] C b,α The correlation between [i′] b,α (i,i′), i,i′=1,2,…,N q ; Then the internal diversity loss L is calculated using the following formula IDL The calculation formula is:
Citation Information
Patent Citations
Video text retrieval method based on differential multi-scale multi-granularity feature fusion
CN116226449A
Cross-modal video text retrieval method, system and equipment and medium
CN116910307A
Video retrieval method and system based on local feature enhancement and modal interaction
CN117609553A
Dam defect image-text cross-modal retrieval method and model
WO2022242388A1
Cited By
Text video cross-modal matching method based on asymmetric semantic optimization
CN119719800A
Text video retrieval method for perceiving importance of spatio-temporal characteristics
CN119884416A
Multi-label extraction and pre-screening text-to-video retrieval method and system
CN119917694A
Partial correlation video retrieval method based on context clue mining
CN121233809A
