A multi-granularity semantic alignment text-video retrieval method based on generated subtitles

Abstract

The application belongs to the field of text-video retrieval, and particularly relates to a multi-granularity semantic alignment text-video retrieval method based on generated subtitles, comprising: establishing an SAEC model, training the same, and performing text-video matching retrieval through the trained SAEC model. A new model SAEC is proposed, the model matches local details in the video and understands the global semantics of the video, and the retrieval accuracy of text to video is significantly improved. Meanwhile, the model can not only understand the direct content of the video, but also grasp more complex situations through the jointly generated subtitles and the video.

Description

Technical Field

[0001] This invention belongs to the field of text-video retrieval, specifically relating to a multi-granularity semantic alignment text-video retrieval method based on generated subtitles. Background Technology

[0002] In recent years, with the significant increase in the number of videos on the Internet, text-based video retrieval—that is, efficiently retrieving target videos through text descriptions—has become increasingly important, possessing significant practical value and research significance. The core issue of text-video retrieval is how to bridge the semantic gap between different modalities of data.

[0003] Over the past few years, large-scale text-image contrastive pre-training has achieved tremendous success and inspiration in multimodal text-visual understanding tasks. A representative example is the CLIP method, which uses a dual-tower structure to directly extract visual and textual features from image pixels and sentences respectively, and then maps them to a common space for end-to-end training. The pre-training task only employs contrastive learning, bringing image-text pairs with similar semantics closer together in the common space and pushing image-text pairs with different semantics further apart, thereby obtaining a unified representation of image and text data. Furthermore, for downstream tasks, CLIP also uses a prompt to adapt the downstream tasks to be consistent with the pre-training task, solving the problem of task misalignment. The CLIP model not only performs well in multimodal retrieval tasks but also shows good generalization ability in traditional unimodal tasks. For example, it outperforms pure vision models in zero-shot image classification on the ImageNet dataset. However, it should be noted that the power of the CLIP model largely stems from its 400 million image and text training datasets, allowing the CLIP model to store a vast amount of semantic knowledge. CLIP trains an image encoder and a text encoder rich in world knowledge based on a dataset of over 400 million image-text pairs. Some research has begun adapting pre-trained text and image encoders to the video domain, achieving significant improvements. While these advancements have shown promising progress on VTR tasks, cross-modal semantic alignment still requires further exploration.

[0004] Meanwhile, in the field of natural language processing, the GPT model based on the Transformer architecture has successfully generated text based on given prompts through training on large-scale text corpora. Recently, ZeroCap proposed a method combining CLIP and GPT-2 language models to generate text descriptions of input images. This method can leverage the capabilities of both models in a completely zero-shot scenario without retraining or fine-tuning the model parameters. A recent study extended ZeroCap's zero-shot capability to the video domain, generating highly semantically generalized captions without any additional training. In this study, we follow their approach, utilizing this video extension capability to generate auxiliary captions without any additional training. In this invention, we borrow this technique to enhance the semantic alignment between text and video.

[0005] This description typically focuses only on specific spatiotemporal segments of a video, providing local semantic information. This stems primarily from the following factors: 1. Limitations of text descriptions: Text descriptions often target a single segment or detail within a video, rather than providing a comprehensive overview of the entire video. 2. Limitations of existing fine-grained matching methods: Existing fine-grained matching methods emphasize the alignment of local information, lacking a comprehensive understanding of the overall semantics of the video. 3. Complexity of video content: The complexity, diversity, and temporal information of video content make the extraction and alignment of global semantics difficult. In contrast, videos present the complete process of an event through a continuous sequence of images, containing rich global visual information. This difference between global and local information can lead to discrepancies in semantic understanding and matching between text and video.

[0006] Most current methods rely on matching specific words in the text with visually apparent content in the video for local alignment. However, although the text description may be highly relevant to a local segment of the video, they may be semantically inconsistent in the global context of the entire video.

[0007] Therefore, although fine-grained matching methods have made significant progress in local alignment, they still have shortcomings in handling the overall semantic expression of videos, which may lead to an inaccurate understanding of the overall theme or context of the video. To address these issues, the following two points need to be considered simultaneously: 1. Integrating global and local perspectives: Limitations of local alignment: Fine-grained matching methods typically match specific objects, actions, or attributes in a video through local alignment, which often relies on video details but ignores the global semantic environment of the video. Combining global and local perspectives: By considering both the global content and local details of the video simultaneously, the problem of missing contextual information that may occur when relying solely on local alignment can be avoided. Integrating global and local perspectives ensures a comprehensive understanding of the video content, thereby improving the relevance and accuracy of retrieval. 2. Better global semantic features: Multimodal information of videos: As a rich carrier of multimodal information, the semantics of videos are not limited to the objects directly presented on the screen, but also include the overall context and storyline constituted by these visual elements. Importance of semantic guidance: Without explicit semantic guidance, video feature aggregation may be interfered with by background noise or irrelevant content. High-level semantic alignment: High-level semantic alignment can better understand and match the overall content and deeper meaning of videos. Therefore, it is necessary to aggregate the global semantic features of videos by generating subtitles or other semantic guidance methods. For example, videos about "economic consulting firms analyzing urban growth" need to be matched based on an understanding of economic activities, rather than just matching specific visual elements. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a multi-granularity semantic alignment text-video retrieval method based on generated subtitles, comprising:

[0009] An SAEC model is established and trained. Text-video matching retrieval is performed using the trained SAEC model. The matching retrieval process includes a global semantic matching (GSM) stage and a local semantic matching (LSM) stage.

[0010] The text-video matching retrieval using the trained SAEC model includes:

[0011] S1: Video subtitle generation;

[0012] S2: The global semantic feature alignment between video and text is completed through the Global Semantic Matching (GSM) stage to achieve matching retrieval;

[0013] S3: Enhance the relevance of search results through the Local Semantic Matching (LSM) stage.

[0014] The beneficial effects of this invention are:

[0015] This invention proposes a novel model, SAEC, which significantly improves the accuracy of text-to-video retrieval by matching local details in videos and understanding their global semantics. Furthermore, by jointly generating captions and video, this invention enables the model to understand not only the direct content of the video but also more complex contexts. Extensive experiments on benchmark datasets demonstrate that this invention effectively combines global and local alignment, comprehensively understands video content, and significantly improves text-to-video retrieval performance. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall structure of the SAEC model of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] A multi-granularity semantic alignment text-video retrieval method based on generated subtitles includes:

[0019] like Figure 1 As shown, an SAEC model is established and trained. Text-video matching retrieval is performed using the trained SAEC model. The matching retrieval process includes a global semantic matching (GSM) stage and a local semantic matching (LSM) stage.

[0020] The text-video matching retrieval using the trained SAEC model includes:

[0021] S1: Video subtitle generation;

[0022] S2: The global semantic feature alignment between video and text is completed through the Global Semantic Matching (GSM) stage to achieve matching retrieval;

[0023] S3: Enhance the relevance of search results through the Local Semantic Matching (LSM) stage.

[0024] Given a set of n videos and its corresponding m subtitles The dataset consists of [various components]. For video v... iIn this invention, the video frames are first sampled to 12 frames using average sampling. A frame encoder, a 12-layer standard Visual Transformer (ViT), is used to process these frames to obtain frame-level features. This invention uses a pre-trained CLIP as the frame encoder. The ViT architecture is similar to the Transformer encoder in Natural Language Processing (NLP), except that ViT introduces a visual tokenization process, transforming the video frames into discrete token sequences. These discrete token sequences are pre-attached with a [CLS] token and then fed into the ViT's Transformer. The [CLS] tokens from the last layer are extracted as frame-level features.

[0025] This invention utilizes a large model to generate captions that highly represent the semantics of videos. It employs CLIP, GPT-2, and their video extensions to achieve zero-shot video caption generation. The core technology is based on a pre-trained GPT-2 language model, which automatically and regressively predicts the next word from an initial prompt. To integrate visual information, a calibrated CLIP loss term (L...) is used. CLIP ) and additional loss terms (L CE This method generates sentences describing the video while maintaining linguistic consistency. Video expansion generates optimized pseudo-tags through iterative sentence generation, uses CLIP to evaluate the similarity between the tags and the video, and adjusts the model to maintain the consistency of video content and linguistic attributes. Furthermore, the system limits each sentence to a maximum of 20 tags and generates subtitles through 30 iterations at a frequency of 3 frames per second, requiring no additional training. The invention then calculates the mean cosine similarity between the generated subtitles and each frame of the corresponding video (each video uses an average sampling of 12 frames), selecting the subtitle with the highest mean cosine similarity as the generated subtitle summarizing the semantics of the video. Subtitle features are then extracted using CLIP's text encoder.

[0026] For a given sentence, this invention directly uses CLIP's text encoder to generate a text representation, which is also initialized by a pre-trained CLIP. Specifically, it is a transformer encoder consisting of a multi-sensor head self-attention and feedforward network. The transformer consists of 12 layers and 8 attention heads. The query, key, and value features have a dimension of 512. The tokenizer used in the experiments is lowercase byte pair encoding (BPE), with a vocab size of 49,152. Before being input into the text encoder, the text token sequence is padded at the beginning and end with [BOS] and [EOS], respectively. The sentence-level (coarse-grained) text features are the [EOS] tokens output from the last layer of the text encoder.

[0027] This invention employs a cross-modal attention layer to leverage caption aggregation video features. In this invention's cross-modal attention layer, a caption is first embedded in c... c ∈R D Projected onto a single query Qc ∈R 1×D Embedding a video frame Projection to key and value In the matrix, D is the potential dimension size of the model of this invention, which is D p The size of the projection dimension is 512. The definition of projection is:

[0028]

[0029] K v =LN(C v W K

[0030] V v =LN(C v W V

[0031] Where LN is the normalization layer, W Q W k and W V yes The projection matrix in the image. To aggregate global video information from subtitles, this invention computes the dot product attention of the query projected subtitle embedding and the video frame embedding.

[0032]

[0033] Next, the important information from the video is fused to obtain the overall semantic features, and the expressive power of the model is further enriched through additional linear layers and residual structures.

[0034] GV = GA(Q) c ,K v V v W O

[0035] r v =LN(GV)+Dropout(W) L LN(GV)))

[0036] Through the above steps, the cross-modal attention layer can effectively utilize the information from the video to integrate subtitles, better capture and utilize complementary information between modalities, and obtain 512-dimensional global semantic features.

[0037] Furthermore, this invention uses combined video and subtitles as new retrieval objects. This invention employs a dynamic routing module that concatenates the fused video features and subtitle features, and then inputs them into the MLP:

[0038]

[0039] The optimized caption features and the aggregated video features are obtained through the dynamic routing module.

[0040] For local semantic matching, this invention uses an adaptive pooling dot product attention module to extract features conditioned on the retrieved text. This invention first projects a text embedding onto a single query, and projects a video frame embedding onto a key-value matrix, where is the latent dimension of the model and is the projection dimension. The projection is defined as:

[0041] Specifically, in the adaptive pooling dot product attention module of this invention, the invention first embeds a text into c. t ∈R D Projected onto a single query Q t ∈R 1×D Embedding a video frame into C v ∈R F×D Project to key Sum In the matrix, D represents the potential dimension of the model of this invention. p It refers to the size of the projection dimension. The definition of projection is:

[0042]

[0043] K v =LN(C v W K

[0044] V v =LN(C v W V

[0045] Where LN is the normalization layer, W Q W k and W V yes The projection matrix in the text. (Note that the local semantic reasoning module and the global semantic extraction module do not share weights.) To learn flexible conditions between a given text and frame, this invention computes dot product attention for the query projected text embedding and the video frame embedding. Dot product attention provides relevance weights from the text to each frame, which this invention uses to aggregate the frame embeddings of the value projection:

[0046]

[0047] Therefore, in the cross-modal interaction of this invention, the text embedding of the query projection is used to find the most relevant frame from the key projection. The value projection embedding represents the context of the video, and based on the text, this invention aggregates the key features of the video as much as possible.

[0048] To embed video into a joint space containing text, this invention applies weights. Project the aggregated video representation from the attention module back to R D From:

[0049] r v|t =LN(Attention(Q) t ,K v V V W O )

[0050] Wherein, the obtained output r v|t This is an aggregated video embedding conditioned on text t. Therefore, this invention can learn this embedding, allowing the text to focus on the frame most semantically similar to it through parameters in the dot product attention. Ultimately, this invention obtains local semantic features derived from the text, i.e., r. L :

[0051] r L =LN(FC(r v|t )+r v|t ) T

[0052] FC is a fully connected network that, together with residual connections, provides additional capacity for more complex inference in the aggregation function of this invention. Finally, this invention obtains 512-dimensional conditional local features.

[0053] The final similarity score between text and video is defined as follows:

[0054] S(t i ,v j ) = Cosine(t i ,r Lj )+λCosine(t i ,r Gj )

[0055] Here, a balanced weight is used. Here, the similarity between the i-th text and the i-th video is represented. Following CLIP, the similarity between texts and videos is measured by calculating cosine similarity. Iterative training and recall ranking are performed using the similarity matrix.

[0056] Given a training batch of B text-video pairs, this invention uses a symmetric InfoNCE as the training objective and represents it as follows:

[0057]

[0058] L=(L t2v +L v2t ) / 2

[0059] In the formula, τ is a learnable scaling parameter whose initial value is obtained from the pre-trained CLIP model.

[0060] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-granularity semantic alignment text-video retrieval method based on generated subtitles, characterized in that, include: An SAEC model is established and trained. Text-video matching retrieval is performed using the trained SAEC model. The matching retrieval process includes a global semantic matching stage and a local semantic matching stage. The text-video matching retrieval using the trained SAEC model includes: S1: Video subtitle generation; S2: The global semantic matching stage aligns the global semantic features of the video and text to achieve matching retrieval; The global semantic matching stage aligns the global semantic features of the video and text to achieve matching retrieval, including: S21: Use a cross-modal attention layer to leverage caption aggregation video features; The projection of the cross-modal attention layer is represented as: K v =LN(C v )W K V v =LN(C v )W V Among them, Q C Let c represent the first query matrix. c This indicates subtitle embedding, T represents matrix transpose, and W represents... Q Indicates querying the projection matrix; C v K represents the frame embedding of the video. v W represents the bond matrix. K V represents the key projection matrix; v W represents a value matrix. V The value projection matrix is ​​represented by LN, which represents the normalization layer. S211: Embed a subtitle in c c ∈R D Projected onto a single query Q c ∈R 1×D Embedding a video frame into C v ∈R F×D Project to key Sum In the matrix, D represents the potential dimension of the SAEC model, set to 512. p Let R be the size of the projection dimension, set to 512. D Let R be a D-dimensional real matrix. 1×D It is a 1×D dimensional real matrix. For F×D p A 3D real matrix, R F×D It is an F×D dimensional real matrix; S212: Calculate the dot product attention of the query projected caption embedding and the video frame embedding; S213: Based on query projection caption embedding and video frame embedding, dot product attention fuses important information from the video to obtain global semantic features, and further enriches the expressive power of the model through additional linear layers and residual structures. S22: Optimize the joint representation of subtitle features and video features through the dynamic routing module, perform global semantic feature alignment between video and text, and realize matching retrieval; S221: The fused video features and subtitle features are concatenated and then input into the MLP to achieve knowledge redistribution and obtain global multimodal features that can represent the overall semantics of the video and subtitles; S3: Enhance the relevance of search results through the local semantic matching stage; Enhancing the relevance of search results through the Local Semantic Matching (LSM) stage includes: S31: Use an adaptive pooling dot product attention module for local semantic matching to enhance the relevance of search results; S311: Embed a text in c t ∈R D Projected onto a single query Q t ∈R 1×D Embedding a video frame into C v ∈R F×D Project to key Sum In the matrix, D represents the latent dimension of the SAEC model. p R is the size of the projection dimension. D Let R be a D-dimensional real matrix. 1×D R is a 1×D dimensional real matrix. F×D It is an F×D dimensional real matrix. For F×D p 3D real matrix; S312: Calculate the dot product attention between the query projected text embedding and the video frame embedding; 313: By applying the projection matrix Project the aggregated video representation from the attention module back to R D This yields the aggregated video embedding conditioned on text t, where, D p ×D-dimensional real matrix; S314: Obtain local semantic features based on the aggregated video embedding.

2. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, Video subtitle generation, including: Using CLIP, GPT2, and their video extensions to generate zero-shot video captions: The pre-trained GPT-2 language model generates subtitles by iterating 30 times at a frequency of 3 frames per second. The model automatically regressively predicts the next word from the initial prompt to obtain video subtitles. Video extension generates optimized pseudo-tags through iterative captioning, setting a limit of up to 20 tags per sentence, using CLIP to evaluate the similarity between the pseudo-tags and the video, and adjusting the model to maintain the consistency of video content and language attributes; The CLIP uses a calibrated CLIP loss term LCLIP and an additional loss term LCE to generate sentences describing the video and maintain linguistic consistency.

3. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, Calculate the dot product attention of the query projected caption embedding and the video frame embedding, including: Among them, GA(Q) c ,K v V v ) represents the dot product attention of the query projection caption embedding and video frame embedding, Q c Represents the query matrix, K v V represents the bond matrix. v Let T denote the value matrix, and D denote the matrix transpose. p Indicates the size of the projection dimension.

4. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, Dot-product attention based on query-projected caption embedding and video frame embedding fuses important video information to obtain global semantic features, and further enriches the model's expressive power through additional linear layers and residual structures, including: By fusing key information from the video, global semantic features can be obtained. GV=GA(Q c ,K v ,V v )W O The expressive power of the model is further enriched by adding additional linear layers and residual structures: r v =LN(GV)+Dropout(W L LN(GV)) Where GV represents global semantic features, GA(Q) c ,K v V v ) represents the dot product attention of the query projection caption embedding and video frame embedding, Q c Represents the query matrix, K v V represents the bond matrix. v W represents a value matrix. O Represents an O-dimensional projection matrix; r v W represents video aggregation features. L Let L represent the L-dimensional projection matrix, LN represent the normalization layer, and Dropout() represent the random deactivation function.

5. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, The fused video features and subtitle features are concatenated and then input into an MLP to achieve knowledge redistribution, obtaining global multimodal features that can represent the overall semantics of the video and subtitles, including: Where, r G Represents global multimodal features, MLP stands for Multilayer Perceptron, r v c represents video aggregation features. c This indicates subtitle embedding, T represents matrix transpose, and W represents... R Let R represent the projection matrix. D Let represent a D-dimensional real matrix, and [.] denote a cascade operation.

6. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, Calculate the dot product attention of the query projected text embedding and the video frame embedding, including: Among them, Attention(Q t ,K v V v ) represents the dot product attention of projected text embedding and video frame embedding, Q t K represents the second query matrix. v V represents the bond matrix. v Let T denote the value matrix, and D denote the matrix transpose. p Indicates the size of the projection dimension.

7. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, By applying weights Project the aggregated video representation from the attention module back to R D The resulting aggregated video embedding, conditioned on text t, includes: r v|t =LN(Attention(Q t ,K v ,V v )W O ) Where, r v|t This represents an aggregated video embedding conditionally based on text t, where Attention(Q) t ,K v V v ) represents the dot product attention of projected text embedding and video frame embedding, Q t K represents the second query matrix. v V represents the bond matrix. v LN represents the value matrix and LN represents the normalization layer.

8. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that, Local semantic features are obtained from the aggregated video embeddings, including: r L =LN(FC(r v|t )+r v|t ) T Where, r L Representing local semantic features, FC stands for fully connected network, LN stands for normalization layer, and r v|t This represents the aggregated video embedding conditioned on text t, where T represents the matrix transpose.

Images