Text and video similarity calculation method and system based on multi-granularity alignment

Abstract

The invention discloses a text and video similarity calculation method and system based on multi-granularity alignment. By deeply analyzing hierarchical features contained in texts and videos, a coarse-to-fine multi-granularity alignment framework is designed, so that more refined and more accurate semantic matching is realized. Especially in the aspect of fine-grained alignment, phrase-level features of the text and object-level features of the video are obtained by aggregating word-level features of the text and block-level features of the video, and fine-grained alignment of the text and the video is realized based on the two features, so that fine-grained semantic alignment is more reasonable, and the accuracy of fine-grained alignment is improved. Therefore, the problems of detail information missing and cross-modal semantic difference existing in text video retrieval are effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal retrieval, and in particular to a method and system for calculating similarity between text and video based on multi-granularity alignment. Background Art

[0002] Due to the rapid development of various video platforms, the Internet is flooded with massive videos. Users' demand for efficient video retrieval is increasing, and the task of text video retrieval has emerged.

[0003] The text-video retrieval method based on multi-granularity alignment aims to achieve multi-level alignment of text and video by aligning the multi-granularity features of text and video respectively. The existing methods can be divided into coarse-grained, fine-grained, cross-granular and hierarchical multi-granularity text-video alignment.

[0004] However, the design of fine-grained alignment in existing multi-granularity alignment text-video retrieval methods uses the alignment of word-level features and block-level features. However, these two features have small granularity and are usually difficult to represent specific semantic content, thus affecting the accuracy of text-video retrieval results.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a method and system for calculating the similarity of text and video based on multi-granularity alignment are provided, aiming to solve the problem that the design of fine-grained alignment in the multi-granularity aligned text video retrieval method in the prior art adopts the alignment of word-level features and block-level features. These two features have small granularity and are difficult to represent specific semantic content, thereby affecting the accuracy of text video retrieval results.

[0007] The technical solution adopted by the present invention to solve the problem is as follows:

[0008] In a first aspect, an embodiment of the present invention further provides a method for calculating similarity between text and video based on multi-granularity alignment, the method comprising:

[0009] Obtaining coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features;

[0010] Aggregating the word-level features of the target text and the block-level features of the target video respectively to obtain the phrase-level features of the target text and the object-level features of the target video;

[0011] Perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0012] The overall similarity between the target text and the target video is determined based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0013] In a second aspect, an embodiment of the present invention further provides a text and video similarity calculation system based on multi-granularity alignment, the system comprising:

[0014] A feature extraction module is used to obtain coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features;

[0015] A feature aggregation module, used to aggregate the word-level features of the target text and the block-level features of the target video, respectively, to obtain the phrase-level features of the target text and the object-level features of the target video;

[0016] A multi-granularity alignment module is used to perform fine-grained alignment according to the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and to perform coarse-grained alignment according to the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0017] A comprehensive analysis module is used to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0018] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-mentioned methods for calculating the similarity of text and video based on multi-granularity alignment.

[0019] Beneficial effects of the present invention: The embodiments of the present invention deeply analyze the hierarchical features contained in text and video, and design a coarse-to-fine multi-granularity alignment framework, thereby achieving more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video, the phrase-level features of the text and the object-level features of the video are obtained, and the fine-grained alignment of the text and the video is achieved based on these two features, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text and video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flowchart of a method for calculating similarity between text and video based on multi-granularity alignment provided in an embodiment of the present invention.

[0022] Figure 2 It is a schematic diagram of a multi-granularity alignment framework of text video based on text enhancement provided by an embodiment of the present invention.

[0023] Figure 3 Schematic diagram of an affinity alignment module provided in an embodiment of the present invention.

[0024] Figure 4 4 is a schematic diagram of a fine-grained aggregation module for videos provided in an embodiment of the present invention.

[0025] Figure 5 is a schematic diagram of a multi-granularity alignment module provided in an embodiment of the present invention.

[0026] Figure 6 It is a module schematic diagram of a text and video similarity calculation system based on multi-granularity alignment provided by an embodiment of the present invention.

[0027] Figure 7 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention discloses a method and system for calculating the similarity of text and video based on multi-granularity alignment. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0029] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0030] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0031] In view of the above-mentioned defects of the prior art, the present invention provides a method for calculating the similarity of text and video based on multi-granularity alignment. By deeply analyzing the hierarchical features contained in the text and video, a multi-granularity alignment framework from coarse to fine is designed to achieve more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video to obtain the phrase-level features of the text and the object-level features of the video, the fine-grained alignment of the text and the video is achieved based on these two features, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text video retrieval.

[0032] like Figure 1 As shown, the method specifically comprises the following steps:

[0033] Step S100, obtaining the coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features.

[0034] Specifically, in the process of matching text and video, the alignment of two different granularities of features, global features and local features, is very critical. Global feature alignment can also be called coarse-grained alignment, and detail feature alignment can also be called fine-grained alignment. Text features can be divided into multiple granularities such as sentences, phrases, and words according to coarse and fine granularity, and video features can be divided into multiple granularities such as videos, frames, objects, and blocks according to coarse and fine granularity. Designing appropriate multi-granular alignment is a key factor in accurately calculating the similarity between text and video, and is also a key factor in accurate text and video retrieval.

[0035] In one implementation, obtaining coarse-grained features and fine-grained features corresponding to the target text and the target video to be matched respectively includes:

[0036] Obtain word-level features and sentence-level features corresponding to the target text through a text encoder;

[0037] The frame-level features and block-level features corresponding to each frame of the target video are obtained through a visual encoder, and a frame-level feature sequence is obtained according to the frame-level features of each frame; a target frame-level feature sequence containing timing information is generated according to the frame-level feature sequence through a temporal encoder; the target frame-level feature sequence is aggregated according to the sentence-level features of the target text through a cross-attention mechanism to obtain the global video features of the target video.

[0038] Specifically, the text is a general description of the video content, which may correspond to a single frame or certain specific frames in the video, so the frame-level features are crucial in the text-video matching process. In addition, the text may also be an overview of the action in the entire video, so the global video features are also indispensable in the text-video matching process. In practical applications, in order to quickly extract the multi-granularity features of text and video, this embodiment will pre-train a text encoder and a visual encoder. The text encoder quickly extracts text features of different granularities, including but not limited to sentence-level features and word-level features. The visual encoder quickly extracts video features of different granularities, including but not limited to frame-level features and block-level features (equivalent to local features of frames). In order to obtain global video features, this embodiment also pre-constructs a temporal encoder, and the temporal encoder obtains a target frame-level feature sequence containing temporal information. Different from the method of performing mean pooling on the frontier time dimension to obtain video features, this embodiment interacts the sentence-level features of the text with the frame-level features of the video through an attention mechanism to obtain global video features.

[0039] For example, the text encoder and visual encoder in CLIP are used as the text encoder and video encoder in this embodiment. Since the original video has a high frame rate, 20-25 frames per second, it takes a lot of computation to process all the video frames. The video is sampled at a frame rate of one frame per second, and the maximum frame length is set to F frames. Videos exceeding this limit can be truncated by appropriate strategies. Therefore, the video is defined as v∈R F×3×H×W , consisting of a frame sequence of F frames v=v1,v2,…,v i ,…,v F , as the video encoder input. Define a given text query t as a word sequence t = {w1,w2,…,w i ,…,w M}, as the input of the text encoder.

[0040] Text Encoder: Based on the given text query, add the [EOS] special tag after the last word and use the CLIP text encoder to obtain the text feature z t The principle of the text encoder is as follows:

[0041] z t =F t (t);

[0042] Among them, z t ∈R M×D represents the word feature of the text, that is, the word tag sequence; M represents the length of the word sequence; D represents the dimension of the word feature. The feature g corresponding to the [EOS] tag output by the last layer of the CLIP text encoder t ∈R D As the global feature of the text, that is, the sentence-level feature.

[0043] Video Encoder: Use CLIP's visual encoder to generate video features and set the number of video frames to F frames. Since CLIP's visual encoder can only process a single image, each frame in the video is used as the input of CLIP's visual encoder to obtain the features corresponding to each frame. The principle of the visual encoder is as follows:

[0044]

[0045] Among them, L v =F×N, where N represents the number of block markers for each frame. The [CLS] marker feature corresponding to each frame is used as the frame-level feature f∈R of the video. F×D , block-level feature z v As the local features of the frame. Through the CLIP visual encoder, the frame-level feature sequence Z is obtained v={f1,f2,…,f F}. In order to obtain global video features, several layers (for example, 4 layers) of lightweight Transformer are added after the video encoder as a temporal encoder, and the temporal information of the video is obtained using position encoding. The principle of the temporal encoder is as follows:

[0046]

[0047] in, Represents the target frame-level feature sequence containing temporal information. Next, the target frame-level feature sequence is aggregated to obtain the global video feature.

[0048] The sentence-level features of the text g t As the query vector Q t =g t , taking the target frame-level sequence features as the key vector Sum value vector Aggregate frame-level features into global video features g through cross-attention v .

[0049]

[0050] In one implementation, the text encoder is trained by a preset affinity alignment module, and the affinity alignment module is used to:

[0051] Acquire training data; wherein the training data includes text and video with corresponding relationships, and auxiliary text description generated based on the video by a video understanding model;

[0052] Inputting the text of the training data into the initial text encoder to obtain sentence-level features of the text; and inputting the auxiliary text description of the training data into the initial description encoder to obtain sentence-level features of the auxiliary text description;

[0053] Obtaining intermediate sentence-level features corresponding to the text and the auxiliary text description respectively through the internal layers of the text encoder and the description encoder;

[0054] A triple contrast loss is calculated based on the two intermediate sentence-level features to train the text encoder; wherein the text encoder shares weights with the description encoder.

[0055] In general, video features contain richer semantic information than text features. Therefore, this embodiment is to expand the semantic information of the text so that the text can be more fully aligned with the rich multi-level video semantics. At the same time, under the guidance of the auxiliary text description, the fine-grained features of the text and the video (such as Figure 2 shown).

[0056] Specifically, in order to enhance the text semantic information, this embodiment generates auxiliary text descriptions for videos in the video library based on a pre-trained video understanding model. This method not only expands the original text data, but also enhances the text's ability to express video content and enriches the text's semantic information. Under the guidance of the auxiliary text description, the fine-grained features of the text and video are better aggregated to achieve a more complete alignment of the text and video semantics. In actual application scenarios, in order to enhance the semantic expression ability of the generated text so that it can achieve a closer semantic alignment with the real manual annotation, this embodiment also designs an affinity alignment module. In certain layers inside the text encoder and the description encoder, the intermediate sentence-level features (also referred to as intermediate global features) of the original text and the generated auxiliary text description can be obtained, and a triple contrast loss is designed to reflect the gap between the original text and the auxiliary text, and the text encoder is optimized by converging the triple contrast loss. And the text encoder and the description encoder share weights, which can ensure that the obtained text features remain consistent and coherent. It should be noted that the affinity alignment module is only used in the training phase of the text encoder and is not used in the actual reasoning phase.

[0057] For example, Figure 3 At certain layers inside the text encoder and description encoder, intermediate sentence-level features of the original text and the generated auxiliary text description can be obtained. and A triplet contrastive loss is designed to optimize the sentence-level features of the text (also known as global features) to obtain closely aligned global features of the original text description and the auxiliary video description.

[0058]

[0059] Where B represents the number of text-video pairs in a small batch during training; As an anchor point, is a positive sample, is a negative sample; As an anchor point, is a positive sample, is a negative sample; t corresponds to the given original text; c corresponds to the auxiliary text description generated based on the video.

[0060] Step S200: Aggregate the word-level features of the target text and the block-level features of the target video respectively to obtain the phrase-level features of the target text and the object-level features of the target video.

[0061] Specifically, the present embodiment has an innovative design for the fine-grained features of text and video. For text, some function words may not contain actual semantic information, while some words usually have specific semantic content in the form of phrases; for video, the block-level features of each frame may only correspond to a partial area of ​​a certain object, and do not represent specific semantic content. Based on this phenomenon, the present embodiment pre-constructs a fine-grained feature aggregation module. After obtaining the word-level features of the text and the block-level features of the video, the word-level features of the text and the block-level features of the video are aggregated through the fine-grained feature aggregation module to obtain features with actual semantic meanings, i.e., the phrase-level features of the text and the object-level features of the video. In the semantic matching process of text and video, there is also a semantic association between the phrases in the text description and the objects in the video, and the alignment of the phrase-level features of the text and the object-level features in the video is also crucial. The present embodiment can more effectively achieve the fine-grained feature alignment of text and video through the fine-grained feature aggregation module.

[0062] In one implementation, the step of respectively aggregating the word-level features of the target text and the block-level features of the target video to obtain the phrase-level features of the target text and the object-level features of the target video includes:

[0063] Acquire the auxiliary text description of the target video through the video understanding model, and acquire the word-level features and sentence-level features of the auxiliary text description of the target video through the description encoder;

[0064] Acquire a common embedding space of the target text and the target video, and generate a unified prompt according to the common embedding space;

[0065] The unified prompt is used as a query vector to interact with the word-level features of the target text; the phrase-level features of the target text are obtained according to the word aggregation features obtained after the interaction; wherein, during the training phase, the word aggregation features corresponding to the training data are interactively generated by the unified prompt corresponding to the training data, the word-level features of the text, and the sentence-level features of the auxiliary text description;

[0066] The unified prompt is used as a query vector to interact with the block-level features of the target video, or the unified prompt is used as a query vector to interact with the block-level features of the target video and the sentence-level features of the auxiliary text description of the target video; the object-level features of the target video are obtained according to the block aggregation features obtained after the interaction.

[0067] Furthermore, the step of generating a unified prompt according to the common embedding space is performed by a preset prompt generation module, and the loss function calculation method of the prompt generation module includes:

[0068] Obtaining a number of predicted unified prompts generated by the prompt generation module;

[0069] The similarities between each of the predicted unified prompts are calculated, and the diversity loss of the prompt generation module is calculated based on all the similarities.

[0070] In summary, this embodiment can obtain the auxiliary text description of the video through the visual understanding model (or visual language model), and use it as an additional text modality to enhance the semantic information of the original text. Based on the given original text and the generated auxiliary text description, a coarse-to-fine multi-granularity alignment learning framework is constructed, and hierarchical alignment is performed based on multi-granularity features such as video-sentence, object-phrase, etc.

[0071] Specifically, although the frame-level features represent the global features of each frame in the video, they are still insufficient in terms of precision. From a semantic point of view, the word-level features of the text cannot be well aligned with the frame-level features of the video. In addition, the block-level features output by the visual encoding are not sufficient to represent semantic features with practical significance. In order to better achieve fine-grained alignment of text and video, this embodiment further aggregates the word-level features of the text and the block-level features of the video, and generates the phrase-level features of the text and the object-level features of the video under the guidance of the unified prompt generated based on the embedding space shared by the two modalities, and designs a fine-grained alignment module based on these two features. Among them, the unified prompt is designed as a learnable embedding, which is implemented based on the complete fine-tuning of CLIP. In addition, the generated auxiliary text description can also improve the fine-grained aggregation module to remove redundant information in the video and generate semantically rich text features.

[0072] It should be noted that during the training process, the paired information of whether the text and video are related is known, while in the inference stage, only the paired information of the auxiliary text description and the video is known, and the relevance of the original text and video is unknown. To prevent information leakage, in the inference stage, the auxiliary text description is not used when the word-level features of the text are aggregated through the fine-grained feature aggregation module.

[0073] For example, in this embodiment, the auxiliary text description c of the video can be obtained through the visual understanding model, and used as an additional text modality to enhance the semantic information of the original text t. The caption encoder and the text encoder share weights to ensure that the obtained text features remain consistent and coherent.

[0074] z c =F t (c);

[0075] Similarly, the feature g corresponding to the [EOS] tag output by the last layer of the CLIP text encoder c ∈R D As the global features of the generated auxiliary text description, that is, sentence-level features.

[0076] In order to focus on the same semantic information during the text and video feature aggregation process, this embodiment generates a unified prompt from the common space (i.e., the feature space shared by the text and video) The aggregation of fine-grained features of text and video is guided by shared learning, where features of different categories in the common space correspond to different semantic information and can generate different unified prompts. At the same time, a diversity loss L is designed. div , so that N q Each unified prompt focuses on different semantic information as much as possible. Intuitively, the similarity between different unified prompts should be 0. Diversity loss L div The calculation formula is as follows:

[0077]

[0078] Among them, u i 、u j represents different unified prompts (i≠j); cos between different prompts represents similarity; diversity loss L div It is used to converge the sum of similarities between different unified prompts so that the similarities between different unified prompts approach 0.

[0079] This embodiment provides two fine-grained feature aggregation methods:

[0080] Method 1: Fine-grained feature aggregation for text: Take the unified prompt u as the query, based on V L The fine-grained aggregation module designed for the Transformer layer is used to realize the interaction between the unified prompt and the word-level features of the text, and the features obtained after the unified prompt interacts with the other word-level features when used as a query vector are used as the aggregated features of the word-level features to represent the phrase-level semantics of the text. If there are multiple unified prompts, each unified prompt will interact with each word-level feature, use its similarity with each word-level feature as a weight, and aggregate the word-level features it focuses on.

[0081]

[0082] Fine-grained feature aggregation for videos: Taking the unified hint u as the query, based on V LThe fine-grained aggregation module designed for the Transformer layer realizes the interaction between the unified prompt and the video block features, and uses the features obtained by the interaction of the unified prompt with the rest of the block-level features when used as a query vector as the aggregated features of the block-level features, representing the object-level semantics of the video. If there are multiple unified prompts, each unified prompt will interact with each block-level feature, using its similarity with each block-level feature as a weight to aggregate the block-level features of its concern.

[0083]

[0084] in,

[0085] Method 2: Text-enhanced fine-grained feature aggregation is used. In order to aggregate the fine-grained features of paired text and video with more similar semantic information, the auxiliary text description generated by the video understanding model is used to assist the aggregation of fine-grained features. Figure 4 As shown, using N L The fine-grained aggregation module is composed of a layer of Transformers. Through the interaction between unified prompts, sentence-level features of auxiliary text descriptions, and word-level features of the original text, the phrase-level features of the text are obtained based on the word aggregation features obtained after the interaction. Through the interaction between unified prompts, sentence-level features of auxiliary text descriptions, and block-level features of the video, the object-level features of the video are obtained based on the block aggregation features obtained after the interaction. Thereby, better aggregation of fine-grained features is achieved. It should be noted that for text, the fine-grained feature aggregation method based on text enhancement is only applicable to the training phase, not to the inference phase, to avoid information leakage.

[0086] The fine-grained feature aggregation of text enhancement is specifically reflected in:

[0087]

[0088]

[0089] In one implementation, the unified prompt is used as a query vector to interact with the block-level features of the target video, and the object-level features of the target video are obtained according to the block-aggregated features obtained after the interaction, including:

[0090] Calculating a similarity score between the sentence-level features of the target text and the frame-level features of the target video, and selecting a number of important frames according to the similarity score;

[0091] The object-level features corresponding to the important frames are used as the object-level features of the target video.

[0092] Specifically, considering that the text may only be aligned with some frames in the video, and in order to reduce the amount of calculation in fine-grained similarity calculation, this embodiment pre-designs a similarity perception module (such as Figure 4 The similarity perception module calculates the similarity between the sentence-level features of the text and the frame-level features of the video, and selects several important frames of the video based on the similarity. In the subsequent fine-grained alignment, only the fine-grained similarity between the phrase-level features of the text and the object-level features of these important frames is calculated.

[0093] For example, in order to further reduce the amount of calculation, this embodiment selects several important frames from all the obtained frames based on the similarity-aware strategy, and only retains the object-level features of these important frames for calculation, wherein the importance of the frame is measured by the similarity score between the sentence-level features of the given original text and the frame-level features of the video.

[0094]

[0095] Among them, S represents the similarity vector between the sentence-level features of the text and the frame features of the video, and s represents the cosine similarity.

[0096] Calculate the sentence-level features g of the text t and frame-level features of the video The similarity scores between them are calculated, and the frames are sorted based on the similarity scores, and the N frames with the highest similarity are selected. f The object-level features corresponding to the frames (i.e., the Top-k frames in the following formula) are used as fine-grained features of the video for subsequent fine-grained alignment with the phrase-level features of the text.

[0097] Sel_idx = Top-k(S);

[0098]

[0099] in, represents all object-level features of the video v before frame selection, Each video contains F frames, each containing N q object-level features. represents all object-level features of the video v after frame selection, Each video contains N F Frames, each frame contains N q object-level features.

[0100] Step S300: Perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score.

[0101] Specifically, this embodiment realizes multi-granular semantic alignment between text and video from coarse to fine. For coarse-grained alignment, after using the attention mechanism in the aforementioned steps to interact the sentence-level features of the text with the frame-level features of the video to obtain the global video features, it is coarsely aligned with the sentence-level features of the text to obtain the coarse-grained similarity score. For fine-grained alignment, after the word-level features of the text and the block-level features of the video are aggregated in the aforementioned steps to obtain the phrase-level features of the text and the object-level features of the video, fine-grained alignment is performed based on these two features to obtain the fine-grained similarity score.

[0102] In one implementation, fine-grained alignment is performed according to the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; coarse-grained alignment is performed according to the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score, including:

[0103] According to the phrase-level features of the target text and the object-level features of the target video, a similarity matrix is ​​calculated by matrix multiplication, and the similarity matrix is ​​aggregated by a Softmax function to obtain a fine-grained similarity score;

[0104] A coarse-grained similarity score is calculated by matrix multiplication based on the sentence-level features of the target text and the global video features of the target video.

[0105] Specifically, Figure 5 As shown in the figure, for coarse-grained alignment, based on the global features of the video and the sentence-level features of the text, the similarity is calculated by matrix multiplication and quantified into a coarse-grained similarity score to achieve coarse-grained alignment of the text and the video. For fine-grained alignment, based on the local object-level features of the video and the phrase-level features of the text, the similarity matrix is ​​calculated by matrix multiplication, and the weights are calculated by Softmax to perform weighted summation, which is aggregated into a fine-grained similarity score to achieve fine-grained alignment of the text and the video. Based on these two similarity scores, the overall similarity between the text and the video can be accurately obtained.

[0106] In one implementation, the steps of fine-grained alignment and coarse-grained alignment are performed by a preset multi-grained alignment module; the multi-grained alignment module includes a coarse-grained alignment module and a fine-grained alignment module; the loss function calculation methods corresponding to the coarse-grained alignment module and the fine-grained alignment module during training include:

[0107] Divide the training data in the training dataset into several small batches of data of the same size;

[0108] Selecting a small batch of data, obtaining a coarse-grained similarity matrix corresponding to the small batch of data through the coarse-grained alignment module; calculating the information entropy loss of coarse-grained contrastive learning according to the coarse-grained similarity matrix; wherein the information entropy loss of the coarse-grained contrastive learning is calculated based on the constraints in the row and column directions of the coarse-grained similarity matrix;

[0109] The fine-grained similarity matrix corresponding to the small batch data is obtained through the fine-grained alignment module; the information entropy loss of fine-grained contrastive learning is calculated according to the fine-grained similarity matrix; wherein the information entropy loss of the fine-grained contrastive learning is calculated based on the constraints in the row and column directions of the fine-grained similarity matrix.

[0110] For example, for coarse-grained alignment: based on the global feature g of the video v ∈R D and the sentence-level features of the text g t ∈R D , calculate its similarity S through matrix multiplication Co , to achieve coarse-grained alignment of text and video.

[0111] S Co =(g t ) T g v ∈R 1 ;

[0112] During the training process, the data in the dataset is processed in batches, each time a small batch of data of size B is processed to obtain a similarity matrix R of size B×B Co , using the information entropy loss (InfoNCE Loss) of contrastive learning, the similarity scores of semantically related text videos in the coarse-grained similarity matrix are higher to optimize the retrieval model.

[0113]

[0114]

[0115]

[0116] Among them, t2v represents the constraint in the row direction, that is, text retrieves video, and v2t represents the constraint in the column direction, that is, video retrieves text. They respectively represent the diagonal elements of the coarse-grained similarity matrix, the i-th row and j-th column elements of the coarse-grained similarity matrix (non-diagonal elements of each row), and the j-th row and i-th column elements of the coarse-grained similarity matrix (non-diagonal elements of each column); the optimization goal is to make the similarity on the diagonal of the matrix higher, that is, the similarity between semantically related text and video is higher, so as to achieve alignment of text and video.

[0117] Towards fine-grained alignment: local object-level features from videos and phrase-level features of the text Use matrix multiplication to calculate the similarity matrix The weights are calculated by Softmax and then weighted summed to aggregate them into an example-level similarity score, i.e., the final fine-grained similarity score, to achieve fine-grained alignment of text and video.

[0118]

[0119] Among them, S Fi(i,j) Represents the i-th row and j-th column element of the fine-grained similarity matrix, thus obtaining a size of N f ×N q The similarity matrix represents the fine-grained similarity score between video v and text t. The similarity matrix is ​​further aggregated to obtain the fine-grained example-level similarity score S″ between video v and text t. Fi .

[0120]

[0121]

[0122] Similarly, during training, given a small batch of data of size B, a fine-grained similarity matrix R of size B×B is obtained through the fine-grained alignment module Fi , using the information entropy loss (InfoNCE Loss) of contrastive learning, the similarity scores of semantically related text videos in the fine-grained similarity matrix are higher to optimize the retrieval model.

[0123]

[0124]

[0125]

[0126] in, They respectively represent the diagonal elements of the fine-grained similarity matrix, the elements of the i-th row and j-th column of the fine-grained similarity matrix (the non-diagonal elements of each row of the fine-grained similarity matrix), and the elements of the j-th row and i-th column of the fine-grained similarity matrix (the non-diagonal elements of each column of the fine-grained similarity matrix).

[0127] Step S400, determining the overall similarity between the target text and the target video according to the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0128] In summary, after obtaining the coarse-grained similarity score and the fine-grained similarity score through the coarse-grained alignment module and the fine-grained alignment module in the aforementioned steps, based on these two similarity scores, the overall similarity between the text and the video can be finally obtained, and mutual retrieval between the text and the video can be achieved based on the overall similarity.

[0129] Specifically, in the actual text video retrieval scenario, the text video retrieval task is mainly divided into two directions. The first direction is text retrieval of videos, which is based on the similarity calculated between a given text query t and a video v∈V, and the videos in the video library V are sorted; the second direction is video retrieval of text, which is based on the similarity calculated between a given video query v and a text t∈T, and the texts in the text library T are sorted. The task of text video retrieval aims to learn a similarity function and calculate the similarity score s(t,v) between a given text t and a video v. Therefore, the similarity calculation method between text and video provided in this embodiment can be well applied to the text video retrieval task.

[0130] For example, given a text t and a video v, the coarse-grained alignment module and the fine-grained alignment module are used to obtain the coarse-grained similarity scores S and S, respectively. Co ∈R 1 and the fine-grained similarity score S″ Fi ∈R 1 Based on these two similarity scores, we can obtain the example-level similarity S of the text video as a whole total ∈R 1 .

[0131] S total =S Co +S″ Fi .

[0132] In one implementation, the text encoder, description encoder, video encoder, prompt generation module, fine-grained aggregation module, and multi-granularity alignment module in the aforementioned steps can be integrated into a multi-granularity cross-modal alignment algorithm model. The objective function of the integrated algorithm model consists of a triple loss function of the affinity alignment module, a diversity loss function of the prompt generation module, a coarse-grained contrastive learning loss function of the coarse-grained alignment module, and a fine-grained contrastive learning loss function of the fine-grained alignment module.

[0133] Specifically expressed as:

[0134] L=L Co +α×L Fi +β×L triplet +γ×L div ;

[0135] Among them, α, β and γ represent the learning weights of the fine-grained contrastive learning loss function, the triplet loss function and the diversity loss function, respectively.

[0136] Furthermore, in the test phase of the algorithm model, the trained model weights are loaded, and through the two stages of feature extraction and fine-grained feature aggregation, the coarse-grained features and fine-grained features of the text, as well as the coarse-grained features and fine-grained features of the video are obtained respectively. The coarse-grained similarity matrix and the fine-grained similarity matrix are obtained through the coarse-grained alignment module and the fine-grained alignment module. The coarse-grained similarity matrix can reflect the coarse-grained similarity score of the current text video, and the fine-grained similarity matrix can reflect the fine-grained similarity score of the current text video. Then, the coarse-grained similarity matrix and the fine-grained similarity matrix are added to obtain the final similarity matrix on the test set. The similarity matrix is ​​a numerical matrix, and the values ​​of the rows in the matrix (or the depth of the block) correspond to the similarity between each text and each video, and the values ​​of the columns (or the depth of the block) correspond to the similarity between each video and each text. Based on the similarity matrix, the final similarity score of the text video is determined, thereby realizing text video retrieval and ensuring the reliability and accuracy of the model in the reasoning phase.

[0137] The innovation of the present invention is:

[0138] (1) Based on the most advanced video understanding model, this paper generates corresponding auxiliary text descriptions for videos as a semantic supplement to the original text, which expands the semantic information of the text and enables better alignment between the semantic information of the text and the rich semantic content of the video. In addition, an affinity comparison module is designed to enhance the semantic ability of the auxiliary text description, so that the semantics of the original text description and the auxiliary text description are closely aligned.

[0139] (2) The present invention proposes a multi-granularity cross-modal alignment framework based on text enhancement, including feature extraction, fine-grained feature aggregation and multi-granularity cross-modal alignment, which can better serve text video retrieval. Specifically, based on the original text and the generated auxiliary text description, the present invention constructs a coarse-to-fine multi-granularity alignment learning framework, performs hierarchical alignment based on multi-granularity features such as video-sentence, object-phrase, etc., to achieve more refined and more accurate text video matching, so as to enhance the text video retrieval performance.

[0140] (3) The present invention utilizes auxiliary text descriptions, under the guidance of unified prompts generated in a common embedding space, to generate phrase-level features of text and object-level features of video through a fine-grained feature aggregation module, and performs frame selection based on a similarity perception module to reduce the amount of fine-grained similarity calculation, thereby achieving more refined and accurate text-video matching.

[0141] Based on the above embodiments, the present invention also provides a text and video similarity calculation system based on multi-granularity alignment, such as Figure 6 As shown, the system comprises:

[0142] The feature extraction module 01 is used to obtain the coarse-grained features and fine-grained features corresponding to the target text and the target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features;

[0143] A feature aggregation module 02 is used to aggregate the word-level features of the target text and the block-level features of the target video to obtain the phrase-level features of the target text and the object-level features of the target video;

[0144] The multi-granularity alignment module 03 is used to perform fine-grained alignment according to the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and perform coarse-grained alignment according to the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0145] The comprehensive analysis module 04 is used to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0146] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 7As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a similarity calculation method for text and video based on multi-granularity alignment is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0147] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0148] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a method for calculating similarity between text and video based on multi-granularity alignment.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0150] In summary, the present invention discloses a method and system for calculating the similarity of text and video based on multi-granularity alignment. By deeply analyzing the hierarchical features contained in text and video, a multi-granularity alignment framework from coarse to fine is designed to achieve more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video, the phrase-level features of the text and the object-level features of the video are obtained, and the fine-grained alignment of the text and the video is achieved based on these two features, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text video retrieval.

[0151] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for calculating similarity between text and video based on multi-granularity alignment, characterized in that: The method comprises: Obtaining coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features; Aggregating the word-level features of the target text and the block-level features of the target video respectively to obtain the phrase-level features of the target text and the object-level features of the target video; Perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score; The overall similarity between the target text and the target video is determined based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

2. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 1, characterized in that: Obtain the coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, including: Obtain word-level features and sentence-level features corresponding to the target text through a text encoder; The frame-level features and block-level features corresponding to each frame of the target video are obtained through a visual encoder, and a frame-level feature sequence is obtained according to the frame-level features of each frame; a target frame-level feature sequence containing timing information is generated according to the frame-level feature sequence through a temporal encoder; the target frame-level feature sequence is aggregated according to the sentence-level features of the target text through a cross-attention mechanism to obtain the global video features of the target video.

3. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 2, characterized in that: The text encoder is trained by a preset affinity alignment module, and the affinity alignment module is used to: Acquire training data; wherein the training data includes text and video having a corresponding relationship, and an auxiliary text description generated based on the video by a video understanding model; Inputting the text of the training data into the initial text encoder to obtain sentence-level features of the text; and inputting the auxiliary text description of the training data into the initial description encoder to obtain sentence-level features of the auxiliary text description; Obtaining intermediate sentence-level features corresponding to the text and the auxiliary text description respectively through the internal layers of the text encoder and the description encoder; A triple contrast loss is calculated based on the two intermediate sentence-level features to train the text encoder; wherein the text encoder shares weights with the description encoder.

4. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 3, characterized in that: The steps of respectively aggregating the word-level features of the target text and the block-level features of the target video to obtain the phrase-level features of the target text and the object-level features of the target video include: Acquire the auxiliary text description of the target video through the video understanding model, and acquire the word-level features and sentence-level features of the auxiliary text description of the target video through the description encoder; Acquire a common embedding space of the target text and the target video, and generate a unified prompt according to the common embedding space; The unified prompt is used as a query vector to interact with the word-level features of the target text; the phrase-level features of the target text are obtained according to the word aggregation features obtained after the interaction; wherein, during the training phase, the word aggregation features corresponding to the training data are interactively generated by the unified prompt corresponding to the training data, the word-level features of the text, and the sentence-level features of the auxiliary text description; The unified prompt is used as a query vector to interact with the block-level features of the target video, or the unified prompt is used as a query vector to interact with the block-level features of the target video and the sentence-level features of the auxiliary text description of the target video; the object-level features of the target video are obtained according to the block aggregation features obtained after the interaction.

5. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 4, characterized in that: The step of generating a unified prompt according to the common embedding space is performed by a preset prompt generation module, and the loss function calculation method of the prompt generation module includes: Obtaining a number of predicted unified prompts generated by the prompt generation module; The similarities between each of the predicted unified prompts are calculated, and the diversity loss of the prompt generation module is calculated based on all the similarities.

6. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 4, characterized in that: The unified prompt is used as a query vector to interact with the block-level features of the target video, and the object-level features of the target video are obtained according to the block aggregation features obtained after the interaction, including: Calculating a similarity score between the sentence-level features of the target text and the frame-level features of the target video, and selecting a number of important frames according to the similarity score; The object-level features corresponding to the important frames are used as the object-level features of the target video.

7. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 1, characterized in that: Performing fine-grained alignment according to the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; A coarse-grained alignment is performed based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score, including: According to the phrase-level features of the target text and the object-level features of the target video, a similarity matrix is ​​calculated by matrix multiplication, and the similarity matrix is ​​aggregated by a Softmax function to obtain a fine-grained similarity score; A coarse-grained similarity score is calculated by matrix multiplication based on the sentence-level features of the target text and the global video features of the target video.

8. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 7, characterized in that: The steps of fine-grained alignment and coarse-grained alignment are performed by a preset multi-grained alignment module; The multi-granularity alignment module includes a coarse-granularity alignment module and a fine-granularity alignment module; The loss function calculation methods corresponding to the coarse-grained alignment module and the fine-grained alignment module during training include: Divide the training data in the training dataset into several small batches of data of the same size; Selecting a small batch of data, obtaining a coarse-grained similarity matrix corresponding to the small batch of data through the coarse-grained alignment module; calculating the information entropy loss of coarse-grained contrastive learning according to the coarse-grained similarity matrix; wherein the information entropy loss of the coarse-grained contrastive learning is calculated based on the constraints in the row and column directions of the coarse-grained similarity matrix; The fine-grained similarity matrix corresponding to the small batch data is obtained through the fine-grained alignment module; the information entropy loss of fine-grained contrastive learning is calculated according to the fine-grained similarity matrix; wherein the information entropy loss of the fine-grained contrastive learning is calculated based on the constraints in the row and column directions of the fine-grained similarity matrix.

9. A text and video similarity calculation system based on multi-granularity alignment, characterized in that: The system comprises: A feature extraction module is used to obtain coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features; A feature aggregation module, used to aggregate the word-level features of the target text and the block-level features of the target video, respectively, to obtain the phrase-level features of the target text and the object-level features of the target video; A multi-granularity alignment module is used to perform fine-grained alignment according to the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and to perform coarse-grained alignment according to the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score; A comprehensive analysis module is used to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the method for calculating the similarity of text and video based on multi-granularity alignment as described in any one of claims 1 to 8 above.

Images