Multi-granularity semantic alignment text-video retrieval method based on subtitle generation

Abstract

The invention belongs to the field of text-video retrieval, and particularly relates to a subtitle generation-based multi-granularity semantic alignment text-video retrieval method, which comprises the following steps of: establishing an SAEC model, training the SAEC model, and carrying out text-video matching retrieval through the trained SAEC model. According to the method, a new model SAEC is provided, the model significantly improves the retrieval accuracy from the text to the video by matching local details in the video and understanding the global semantics of the video, and meanwhile, the model not only can understand the direct content of the video, but also can grasp a more complex situation by jointly generating the subtitles and the video.

Description

Technical Field

[0001] The invention belongs to the field of text-video retrieval, and in particular relates to a multi-granularity semantic alignment text-video retrieval method based on generated subtitles. Background Art

[0002] In recent years, with the significant growth of the number of videos on the Internet, text-based video retrieval, that is, efficiently retrieving target videos through text descriptions, has become increasingly important and has great practical value and research significance. The core issue of the text-video retrieval task is how to solve the semantic gap between different modal data.

[0003] In the past few years, large-scale text-image contrast pre-training has achieved great success in multimodal text-visual understanding tasks. For example, the representative CLIP method uses a dual-tower structure to directly extract visual and text features from image pixels and sentences respectively, and then maps them to the common space for end-to-end training. The pre-training task only uses the contrastive learning method to bring image-text pairs with similar semantics closer in the common space and push image-text pairs with different semantics farther away, so as to obtain a unified representation of image and text data. In addition, for downstream tasks, CLIP also adapts downstream tasks to the same way as pre-training tasks through prompts, solving the problem of task misalignment. The CLIP model not only has obvious effects on multimodal retrieval tasks, but also has good generalization capabilities on traditional single-modal tasks. For example, the zero-shot image classification task on the ImageNet dataset exceeds the pure visual model. However, it should be noted that the power of the CLIP model comes more from the 400 million image and text training data, which allows the CLIP model to store a large amount of semantic knowledge. CLIP trained an image encoder and a text encoder with rich world knowledge based on a dataset of more than 400 million image-text pairs. Some research works have begun to adapt pre-trained text and image encoders to the video domain, achieving significant improvements. Although they have significantly improved the VTR task and shown promising progress, cross-modal semantic alignment still needs further exploration.

[0004] Meanwhile, in the field of natural language processing, the GPT model based on the Transformer architecture has successfully generated text based on a given prompt by training on a large-scale text corpus. Recently, ZeroCap proposed a method combining CLIP and the GPT-2 language model to generate a text description of an input image, which is able to leverage the capabilities of both models in a completely zero-shot manner without retraining or fine-tuning the model parameters. A recent study extended ZeroCap's zero-shot capabilities to the video domain, generating subtitles that highly summarize the semantics of the video without any additional training. In this study, the present invention follows them and uses this method to generate auxiliary subtitles using the video extension function without any additional training. In the present invention, the present invention borrows this technology to increase the semantic alignment of text and video.

[0005] This description usually only focuses on a specific spatiotemporal segment of the video and provides local semantic information. This is mainly due to the following points: 1. Limitations of text description: Text descriptions are often targeted at a certain segment or detail in the video, rather than a comprehensive summary of the entire video. 2. Limitations of existing fine-grained matching methods: Existing fine-grained matching methods focus on the alignment of local information and lack a comprehensive understanding of the overall semantics of the video. 3. Complexity of video content: The complexity, diversity, and timing information of video content make it difficult to extract and align global semantics. In contrast, the video presents the complete process of the entire event through a continuous sequence of pictures, which contains rich global visual information. This difference between global and local information may lead to deviations in the semantic understanding and matching process between text and video.

[0006] Most current methods rely on local alignment by matching specific words in the text to visually visible content in the video. However, although the text description may be highly relevant to a local segment of the video, they may be semantically inconsistent in the global context of the entire video.

[0007] Therefore, although fine-grained matching methods have made significant progress in local alignment, they still have shortcomings in dealing with the overall semantic expression of videos, which may lead to an inability to accurately understand the overall theme or context of the video. To address these issues, the following two points need to be considered simultaneously: 1. Integrate global and local perspectives: Limitations of local alignment: Fine-grained matching methods usually match specific objects, actions or attributes in videos through local alignment, which often relies on the details of the video but ignores the global semantic environment of the video. Combination of global and local: By considering the global content and local details of the video at the same time, the problem of missing contextual information that may occur when relying solely on local alignment can be avoided. Integrating global and local perspectives ensures a comprehensive understanding of the video content, thereby improving the relevance and accuracy of retrieval. 2. Better global semantic features: Multimodal information of videos: As a rich multimodal information carrier, the semantics of videos are not limited to the objects directly presented in the picture, but also include the overall context and storyline composed of these visual elements. The importance of semantic guidance: Without clear semantic guidance, video feature aggregation may be disturbed by background noise or irrelevant content. High-level semantic alignment: High-level semantic alignment can better understand and match the overall content and deep meaning of the video. Therefore, it is necessary to aggregate the global semantic features of the video by generating subtitles or other semantic guidance methods. For example, the video of "Economic Consulting Company Analyzing Urban Growth" needs to be matched based on the understanding of economic activities, rather than just matching by specific visual elements. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a multi-granularity semantic alignment text-video retrieval method based on generated subtitles, comprising:

[0009] A SAEC model is established and trained, and text-video matching retrieval is performed using the trained SAEC model, wherein the matching retrieval process includes a global semantic matching (GSM) stage and a local semantic matching (LSM) stage;

[0010] The text-video matching retrieval is performed using the trained SAEC model, including:

[0011] S1: Video subtitle generation;

[0012] S2: The global semantic feature alignment of video and text is completed through the global semantic matching (GSM) stage to achieve matching retrieval;

[0013] S3: Enhance the relevance of retrieval results through the local semantic matching (LSM) stage.

[0014] Beneficial effects of the present invention:

[0015] This paper proposes a new model, SAEC, which significantly improves the accuracy of text-to-video retrieval by matching local details in the video and understanding the global semantics of the video. At the same time, by jointly generating subtitles and videos, the present invention enables the model to not only understand the direct content of the video, but also grasp more complex contexts. Extensive experiments on benchmark datasets have been conducted, demonstrating that the present invention effectively combines global and local alignment, fully understands the video content, and significantly improves text-to-video retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the overall structure of the SAEC model of the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] A multi-granular semantically aligned text-video retrieval method based on generated subtitles, comprising:

[0019] like Figure 1 As shown, a SAEC model is established and trained, and text-video matching retrieval is performed through the trained SAEC model, and the matching retrieval process includes a global semantic matching (GSM) stage and a local semantic matching (LSM) stage;

[0020] The text-video matching retrieval is performed using the trained SAEC model, including:

[0021] S1: Video subtitle generation;

[0022] S2: The global semantic feature alignment of video and text is completed through the global semantic matching (GSM) stage to achieve matching retrieval;

[0023] S3: Enhance the relevance of retrieval results through the local semantic matching (LSM) stage.

[0024] Given a video consisting of n and its corresponding m subtitles For the video v i∈V, the present invention first samples the video frames by 12 frames using average sampling. The frame encoder is used to process these frames to obtain frame-level features, which is a 12-layer standard visual transformer (ViT). The present invention uses pre-trained CLIP as a frame encoder. The architecture of ViT is the same as the Transformer encoder in natural language processing (NLP), except that ViT introduces a visual tokenization process to convert video frames into discrete token sequences. The discrete token sequence is pre-attached with a [CLS] token and then fed into the Transformer of ViT. The [CLS] token of the last layer is extracted as a frame-level feature.

[0025] The present invention uses a large model to generate subtitles that can highly express the semantics of the video. The present invention uses CLIP, GPT2 and its video extension to achieve zero-shot video subtitle generation. The core technology is based on a pre-trained GPT-2 language model, which automatically regressively predicts the next word from the initial prompt. In order to integrate visual information, a calibrated CLIP loss term (L CLIP ) and an additional loss term (L CE ) to generate sentences describing the video and maintain language consistency. Video extension optimizes pseudo-labels by iterative sentence generation, uses CLIP to evaluate the similarity between the labels and the video, and adjusts the model to maintain the consistency and language properties of the video content. In addition, the system sets a limit of up to 20 labels per sentence, and generates subtitles by performing 30 iterations at a frequency of 3 frames per second without additional training. The present invention then calculates the average cosine similarity between the generated subtitles and each frame of the corresponding video (each video uses an average sampling of 12 frames), and selects the subtitles with the highest average cosine similarity as the generated subtitles that summarize the semantics of the video. Subtitle features are extracted through the text encoder of CLIP.

[0026] For a given sentence, the present invention directly uses the text encoder of CLIP to generate a text representation, which is also initialized by the pre-trained CLIP. Specifically, it is a transformer encoder consisting of a multi-sensor head self-attention and feedforward network. The transformer consists of 12 layers and 8 attention heads. The dimension of the query, key and value features is 512. The tagger used in the experiment is lowercase byte pair encoding (BPE) with a vocab size of 49 152. Before being input into the text encoder, the text token sequence is padded with [BOS] and [EOS] at the beginning and end, respectively. The sentence-level (coarse-grained) text feature is the [EOS] token output by the last layer of the text encoder.

[0027] The present invention uses a cross-modal attention layer to aggregate video features using subtitles. In the cross-modal attention layer of the present invention, the present invention first embeds a subtitle into c c ∈R D Projected into a single query Qc ∈R 1×D , embed a video frame into Projection to keys and values In the matrix, D is the potential dimension size of the model of the present invention, and D p The size of the projection dimension is 512. The projection is defined as:

[0028]

[0029] K v =LN(C v )W K

[0030] V v =LN(C v )W V

[0031] Where LN is the normalization layer, W Q ,W k and W V yes In order to aggregate the global information of the video by the subtitles, the present invention will calculate the dot product attention of the query projected subtitle embedding and the video frame embedding.

[0032]

[0033] Then, the important information of the video is integrated to obtain the overall semantic features, and the expression ability of the model is further enriched through additional linear layers and residual structures.

[0034] GV=GA(Q c ,K v ,V v )W O

[0035] r v =LN(GV)+Dropout(W L LN(GV)))

[0036] Through the above steps, the cross-modal attention layer can effectively use the information integrated from the video by subtitles, better capture and utilize the complementary information between modalities, and obtain 512-dimensional global semantic features.

[0037] Furthermore, the present invention uses the combined video and subtitles as a new retrieval object. The present invention uses a dynamic routing module to concatenate the fused video features and subtitle features and then input them into the MLP:

[0038]

[0039] The joint representation of optimized subtitle features and aggregated video features is obtained through the dynamic routing module.

[0040] For local semantic matching, the present invention uses an adaptive pooling dot product attention module to extract features that retrieve text as a condition. The present invention first projects a text embedding to a single query and a video frame embedding to a key and value matrix, where is the potential dimension size of the present invention's model and is the size of the projection dimension. The projection is defined as:

[0041] Specifically, in the adaptive pooling dot product attention module of the present invention, the present invention first embeds a text into c t ∈R D Projected into a single query Q t ∈R 1×D , embed a video frame into C v ∈R F×D Projection to Key Sum In the matrix, D is the potential dimension size of the model of the present invention, D p is the size of the projection dimension. The projection is defined as:

[0042]

[0043] K v =LN(C v )W K

[0044] V v =LN(C v )W V

[0045] Where LN is the normalization layer, W Q ,W k and W V yes The projection matrix in . (Note that the local semantic reasoning module and the global semantic extraction module do not share weights) In order to learn flexible conditions between given text and frames, the present invention will calculate the dot product attention of the query projected text embedding and the video frame embedding. The dot product attention provides the relevance weights from the text to each frame, and the present invention uses these weights to aggregate the frame embedding of the value projection:

[0046]

[0047] Therefore, in the cross-modal interaction of the present invention, the text embedding of the query projection is used to find the most relevant frame from the key projection. The value projection embedding represents the context of the video, and based on the text, the present invention aggregates the key features of the video as much as possible.

[0048] In order to embed the video into a joint space with text, the present invention applies weights Project the aggregated video representation in the attention module back to R D In the result:

[0049] r v|t =LN(Attention(Q t ,K v ,V V )W O )

[0050] The output r v|t is the aggregated video embedding conditioned on the text t. Thus, the present invention can learn this embedding so that the text can focus on the frames that are most similar to its semantics through the parameters in the dot product attention. Finally, the present invention obtains the local semantic features obtained from the text, namely r L :

[0051] r L =LN(FC(r v|t )+r v|t ) T

[0052] FC is a fully connected network, which together with the residual connection provides additional capacity for more complex reasoning in the aggregation function of the present invention. Finally, the present invention obtains 512-dimensional conditional local features.

[0053] The final similarity score of text and video is defined as:

[0054] S(t i ,v j )=Cosine(t i ,r Lj )+λCosine(t i ,r Gj )

[0055] Where is a balance weight. Where represents the similarity between the th text and the th video. Following CLIP, the similarity between text and video is measured by calculating cosine cosine similarity. Iterative training and recall ranking are performed through the similarity matrix.

[0056] Given a training batch of B text-video pairs, the present invention uses the symmetric InfoNCE as the training target and expresses it as:

[0057]

[0058] L=(L t2v +L v2t ) / 2

[0059] Where τ is a learnable scaling parameter whose initial value is obtained from the pre-trained CLIP model.

[0060] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-granular semantically aligned text-video retrieval method based on generated subtitles, characterized in that: include: A SAEC model is established and trained, and text-video matching retrieval is performed using the trained SAEC model, wherein the matching retrieval process includes a global semantic matching (GSM) stage and a local semantic matching (LSM) stage; The text-video matching retrieval is performed using the trained SAEC model, including: S1: Video subtitle generation; S2: The global semantic feature alignment of video and text is completed through the global semantic matching (GSM) stage to achieve matching retrieval; The global semantic matching (GSM) stage is used to align the global semantic features of the video and text, and achieve matching retrieval, including: S21: Aggregate video features using subtitles using cross-modal attention layers:; S211: Embed a subtitle into c c ∈R D Projected into a single query Q c ∈R 1×D , embed a video frame into Projection to keys and values In the matrix, D is the potential dimension size of the SAEC model, which is set to 512, and D p is the projection dimension size, set to 512, R D is a D-dimensional real matrix, R 1×D is a 1×D dimensional real matrix, F×D p dimensional real matrix; S212: Calculate the dot product attention of the query projected subtitle embedding and the video frame embedding; S213: Based on the dot product attention of query projected subtitle embedding and video frame embedding, the important information of the video is fused to obtain global semantic features, and the expression ability of the model is further enriched through additional linear layers and residual structures; S22: Optimize the joint representation of subtitle features and video features through the dynamic routing module, align the global semantic features of video and text, and realize matching retrieval; S221: The fused video features and subtitle features are connected in series and then input into the MLP to achieve knowledge redistribution and obtain global multimodal features that can represent the overall semantics of the video and subtitles; S3: Enhance the relevance of retrieval results through the local semantic matching (LSM) stage.

2. A multi-granular semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: Video subtitle generation, including: Zero-shot video caption generation using CLIP, GPT2, and its video extensions: Captions are generated based on the pre-trained GPT-2 language model through 30 iterations at a frequency of 3 frames per second. The model automatically regressively predicts the next word from the initial prompt to obtain video captions; Video expansion optimizes pseudo-labels through iterative captioning, sets a limit of up to 20 labels per sentence, uses CLIP to evaluate the similarity of pseudo-labels to videos, and adjusts the model to maintain consistency of video content and language properties; The CLIP adopts a calibrated CLIP loss term (LCLIP) and an additional loss term (LCE) to generate sentences describing the video and maintain language consistency.

3. A multi-granular semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: The projection of the cross-modal attention layer is expressed as: K v =LN(C v )W K V v =LN(C v )W V Among them, Q C represents the first query matrix, c c represents subtitle embedding, T represents matrix transposition, W Q represents the query projection matrix; K v represents the frame embedding of the video, C v represents the video feature matrix, W K represents the key projection matrix; V v represents the key and value matrix, W V represents the value projection matrix and LN represents the normalization layer.

4. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: Compute the dot product attention of the query projected caption embedding and the video frame embedding, including: Among them, GA(Q c , K v , V v ) represents the dot product attention of query projected caption embedding and video frame embedding, Q c represents the query matrix, K v represents the frame embedding of the video, V v represents the key and value matrix, T represents the matrix transpose, D p Indicates the size of the projection dimension.

5. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: The dot product attention based on query projection subtitle embedding and video frame embedding fuses important information of the video to obtain global semantic features, and further enriches the expression ability of the model through additional linear layers and residual structures, including: Fusion of important video information to obtain global semantic features: GV=GA(Q c ,K v ,V v )W O The expressive power of the model is further enriched through additional linear layers and residual structures: r v =LN(GV)+Dropout(W L LN(GV))) Among them, GV represents the global semantic feature, GA(Q c , K v , V v ) represents the dot product attention of query projected caption embedding and video frame embedding, Q c represents the query matrix, K v represents the frame embedding of the video, V v represents the key and value matrix, W O represents the O-dimensional projection matrix; r v represents the video aggregation feature, W L represents the L-dimensional projection matrix, LN represents the normalization layer, and Dropout() represents the random dropout function.

6. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: The fused video features and subtitle features are concatenated and then input into the MLP to achieve knowledge redistribution and obtain global multimodal features that can represent the overall semantics of the video and subtitles, including: Among them, r G represents global multimodal features, MLP represents multi-layer perceptron, r v represents the video aggregation feature, c c represents subtitle embedding, T represents matrix transposition, W R represents the projection matrix, R D represents a D-dimensional real matrix, and [.] represents a cascade operation.

7. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 1, characterized in that: The relevance of retrieval results is enhanced through the Local Semantic Matching (LSM) stage, including: S31: Use adaptive pooling dot product attention module for local semantic matching to enhance the relevance of retrieval results; S311: Embed a text into c t ∈R D Projected into a single query Q t ∈R 1×D , embed a video frame into C v ∈R F×D Projection to Key Sum In the matrix, D is the potential dimension size of the SAEC model, and D p is the projection dimension size, R D is a D-dimensional real matrix, R 1×D is a 1×D dimensional real matrix, R F×D is a real matrix of dimension F×D, F×D p dimensional real matrix; S312: Calculate the dot product attention of the query projection text embedding and the video frame embedding; 313: By applying the projection matrix Project the aggregated video representation in the attention module back to R D , we get the aggregated video embedding conditioned on the text t, where D p ×D-dimensional real matrix; S314: Obtain local semantic features based on the aggregated video embedding.

8. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 7, characterized in that: Compute the dot product attention of the query projected text embedding and the video frame embedding, including: Among them, Attention(Q t , K v , V v ) represents the dot product attention of the projected text embedding and the video frame embedding, Q t represents the second query matrix, K v represents the frame embedding of the video, V v represents the key and value matrix, T represents the matrix transpose, D p Indicates the size of the projection dimension.

9. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 7, characterized in that: By applying weight Project the aggregated video representation in the attention module back to R D , we get the aggregated video embedding conditioned on the text t, including: r v|t =LN(Attention(Q t ,K v ,V v )W O ) Among them, r v|t represents the aggregated video embedding conditioned on the text t, Attention(Q t , K v , V v ) represents the dot product attention of the projected text embedding and the video frame embedding, Q t represents the second query matrix, K v represents the frame embedding of the video, V v represents the key and value matrix and LN represents the normalization layer.

10. The multi-granularity semantic alignment text-video retrieval method based on generated subtitles according to claim 7, characterized in that: Obtain local semantic features based on aggregated video embedding, including: r L =LN(FC(r v|t )+r v|t ) T Among them, r L represents local semantic features, FC represents a fully connected network, LN represents a normalization layer, and r v|t represents the aggregated video embedding conditioned on the text t, and T represents the matrix transpose.

Images