Method and system for calculating similarity between text and video based on multi-granularity alignment

Abstract

The present invention discloses a method and system for calculating the similarity of text and video based on multi-granularity alignment. By deeply analyzing the hierarchical features contained in text and video, a multi-granularity alignment framework from coarse to fine is designed to achieve more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video, the phrase-level features of the text and the object-level features of the video are obtained. Based on these two features, fine-grained alignment of the text and the video is achieved, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text and video retrieval.

Description

Technical Field

[0001] The present invention relates to the field of cross-modal retrieval technology, and in particular to a method and system for calculating similarity between text and video based on multi-granularity alignment. Background Art

[0002] Due to the rapid development of various video platforms, the Internet is flooded with massive videos. Users' demand for efficient video retrieval is increasing, and the task of text video retrieval has emerged.

[0003] The text-video retrieval method based on multi-granularity alignment aims to achieve multi-level alignment of text and video by aligning the multi-granularity features of text and video respectively. The existing methods can be divided into coarse-grained, fine-grained, cross-granularity and hierarchical multi-granularity text-video alignment.

[0004] However, the fine-grained alignment design in existing multi-granularity alignment text-video retrieval methods uses the alignment of word-level features and block-level features. However, these two features have small granularity and are usually difficult to represent specific semantic content, thus affecting the accuracy of text-video retrieval results.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0006] The technical problem to be solved by the present invention is that, in response to the above-mentioned defects of the prior art, a method and system for calculating the similarity of text and video based on multi-granularity alignment is provided, aiming to solve the problem that the design of fine-grained alignment in the multi-granularity aligned text and video retrieval method in the prior art adopts the alignment of word-level features and block-level features. These two features have small granularity and are difficult to represent specific semantic content, thereby affecting the accuracy of text and video retrieval results.

[0007] The technical solutions adopted by the present invention to solve the problem are as follows:

[0008] In a first aspect, an embodiment of the present invention further provides a method for calculating similarity between text and video based on multi-granularity alignment, the method comprising:

[0009] Obtaining coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features;

[0010] Aggregating the word-level features of the target text and the block-level features of the target video respectively to obtain phrase-level features of the target text and object-level features of the target video;

[0011] Performing fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; performing coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0012] The overall similarity between the target text and the target video is determined based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0013] In a second aspect, an embodiment of the present invention further provides a text and video similarity calculation system based on multi-granularity alignment, the system comprising:

[0014] A feature extraction module is used to obtain coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively. The fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features.

[0015] A feature aggregation module, configured to aggregate the word-level features of the target text and the block-level features of the target video to obtain phrase-level features of the target text and object-level features of the target video;

[0016] A multi-granularity alignment module is configured to perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and to perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0017] A comprehensive analysis module is used to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to implement the retrieval of videos in the video library given a text; or to implement the retrieval of texts in the text library given a video.

[0018] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor to implement any of the steps of the above-mentioned method for calculating the similarity of text and video based on multi-granularity alignment.

[0019] Beneficial effects of the present invention: The embodiments of the present invention deeply analyze the hierarchical features contained in text and video, and design a multi-granularity alignment framework from coarse to fine, thereby achieving more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video to obtain the phrase-level features of the text and the object-level features of the video, fine-grained alignment of the text and the video is achieved based on these two features, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text and video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 4 is a flow chart of a method for calculating similarity between text and video based on multi-granularity alignment provided by an embodiment of the present invention.

[0022] Figure 2 3 is a schematic diagram of a multi-granularity alignment framework of text videos based on text enhancement provided by an embodiment of the present invention.

[0023] Figure 3 Schematic diagram of the affinity alignment module provided in an embodiment of the present invention.

[0024] Figure 4 4 is a schematic diagram of a fine-grained video aggregation module provided by an embodiment of the present invention.

[0025] Figure 5 Schematic diagram of a multi-granularity alignment module provided by an embodiment of the present invention.

[0026] Figure 6 3 is a module diagram of a text and video similarity calculation system based on multi-granularity alignment provided by an embodiment of the present invention.

[0027] Figure 7 This is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention discloses a method and system for calculating text and video similarity based on multi-granularity alignment. To make the objectives, technical solutions, and effects of the present invention more clear and explicit, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention.

[0029] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0030] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0031] In response to the above-mentioned defects of the prior art, the present invention provides a method for calculating the similarity of text and video based on multi-granularity alignment. By deeply analyzing the hierarchical features contained in text and video, a multi-granularity alignment framework from coarse to fine is designed to achieve more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video, the phrase-level features of the text and the object-level features of the video are obtained, and based on these two features, the fine-grained alignment of the text and the video is achieved, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text and video retrieval.

[0032] like Figure 1 As shown, the method specifically includes the following steps:

[0033] Step S100: Obtain the coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, wherein the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features.

[0034] Specifically, in the text and video matching process, aligning features of different granularities—global and local features—is crucial. Global feature alignment is also called coarse-grained alignment, while detailed feature alignment is also called fine-grained alignment. Text features can be categorized into multiple levels of granularity, such as sentences, phrases, and words, based on coarse and fine granularity. Video features can be categorized into multiple levels of granularity, such as videos, frames, objects, and blocks, based on coarse and fine granularity. Designing appropriate multi-granularity alignment is crucial for accurately calculating the similarity between text and video, and is also crucial for accurate text and video retrieval.

[0035] In one implementation, obtaining coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, includes:

[0036] Obtain word-level features and sentence-level features corresponding to the target text through a text encoder;

[0037] The frame-level features and block-level features corresponding to each frame of the target video are obtained through a visual encoder, and a frame-level feature sequence is obtained based on the frame-level features of each frame; a target frame-level feature sequence containing timing information is generated based on the frame-level feature sequence through a temporal encoder; the target frame-level feature sequence is aggregated according to the sentence-level features of the target text through a cross-attention mechanism to obtain the global video features of the target video.

[0038] Specifically, the text is a general description of the video content, which may correspond to a single frame or certain specific frames in the video, so frame-level features are crucial in the text-video matching process. In addition, the text may also be an overview of the action in the entire video, so global video features are also indispensable in the text-video matching process. In practical applications, in order to quickly extract multi-granularity features of text and video, this embodiment will pre-train a text encoder and a visual encoder. The text encoder is used to quickly extract text features of different granularities, including but not limited to sentence-level features and word-level features. The visual encoder is used to quickly extract video features of different granularities, including but not limited to frame-level features and block-level features (equivalent to local features of frames). In order to obtain global video features, this embodiment also pre-constructs a temporal encoder, through which a target frame-level feature sequence containing temporal information is obtained. Different from the method of performing mean pooling on the frontier time dimension to obtain video features, this embodiment uses an attention mechanism to interact the sentence-level features of the text with the frame-level features of the video to obtain global video features.

[0039] For example, the text encoder and visual encoder in CLIP are used as the text encoder and video encoder in this embodiment. Since the original video has a high frame rate of 20-25 frames per second, processing all video frames requires a very large amount of computation. The video is sampled at a frame rate of one frame per second and the maximum frame length is set to F frames. Videos exceeding this limit can be truncated using appropriate strategies. Therefore, the video is defined as v∈R F×3×H×W , consisting of a frame sequence of F frames v=v1,v2,…,v i ,…,v F , as the video encoder input. Define the given text query t as a word sequence t={w1,w2,…,w i ,…,w M}, as input to the text encoder.

[0040] Text Encoder: Based on a given text query, add a special [EOS] tag after the last word and use CLIP's text encoder to obtain text features z t The principle of the text encoder is as follows:

[0041] z t =F t (t);

[0042] Among them, z t ∈R M×D Represents the word features of the text, that is, the word tag sequence; M represents the length of the word sequence; D represents the dimension of the word feature. The feature g corresponding to the [EOS] tag output by the last layer of the CLIP text encoder is t ∈R D As the global feature of the text, that is, the sentence-level feature.

[0043] Video Encoder: Use CLIP's visual encoder to generate video features and set the number of video frames to F frames. Since CLIP's visual encoder can only process a single image, each frame in the video is used as input to the CLIP visual encoder to obtain the corresponding features for each frame. The principle of the visual encoder is as follows:

[0044]

[0045] Among them, L v =F×N, where N represents the number of block labels for each frame. The [CLS] label feature corresponding to each frame is used as the frame-level feature f∈R of the video. F×D , block-level feature z v As the local features of the frame. Through the CLIP visual encoder, the frame-level feature sequence Z is obtained v={f1,f2,…,f F To obtain global video features, several layers (e.g., 4 layers) of lightweight Transformer are added after the video encoder as a temporal encoder, and positional encoding is used to obtain the temporal information of the video. The principle of the temporal encoder is as follows:

[0046]

[0047] in, Represents the target frame-level feature sequence containing temporal information. Next, the target frame-level feature sequence is aggregated to obtain the global video feature.

[0048] The sentence-level feature g of the text t As the query vector Q t =g t , taking the target frame-level sequence features as the key vector Sum value vector Aggregate frame-level features into global video features g through cross-attention v .

[0049]

[0050] In one implementation, the text encoder is trained by a preset affinity alignment module, and the affinity alignment module is used to:

[0051] Obtain training data; wherein the training data includes text and video with corresponding relationships, and auxiliary text descriptions generated based on the video by a video understanding model;

[0052] Inputting the text of the training data into the initial text encoder to obtain sentence-level features of the text; and inputting the auxiliary text description of the training data into the initial description encoder to obtain sentence-level features of the auxiliary text description;

[0053] Obtaining, through the internal layers of the text encoder and the description encoder, intermediate sentence-level features corresponding to the text and the auxiliary text description, respectively;

[0054] A triplet contrast loss is calculated based on the two intermediate sentence-level features to train the text encoder; wherein the text encoder shares weights with the description encoder.

[0055] In general, video features contain richer semantic information than text features. Therefore, this embodiment aims to expand the semantic information of text so that text can be more fully aligned with the rich multi-level video semantics. At the same time, under the guidance of auxiliary text description, fine-grained features of text and video (such as Figure 2 shown).

[0056] Specifically, in order to enhance the semantic information of the text, this embodiment generates auxiliary text descriptions for videos in the video library based on a pre-trained video understanding model. This approach not only expands the original text data, but also enhances the text's ability to express video content and enriches the text's semantic information. Under the guidance of the auxiliary text description, the fine-grained features of the text and video are better aggregated to achieve a more complete alignment of the text and video semantics. In actual application scenarios, in order to enhance the semantic expression ability of the generated text and enable it to achieve a closer semantic alignment with the real manual annotation, this embodiment also designs an affinity alignment module. In certain layers within the text encoder and description encoder, intermediate sentence-level features (also called intermediate global features) of the original text and the generated auxiliary text description can be obtained, and a triple contrast loss is designed to reflect the gap between the original text and the auxiliary text. The text encoder is optimized by converging the triple contrast loss. In addition, the text encoder and the description encoder share weights, which can ensure that the obtained text features remain consistent and coherent. It should be noted that this affinity alignment module is only used in the training phase of the text encoder and is not used in the actual reasoning phase.

[0057] For example, Figure 3 At certain layers within the text encoder and description encoder, intermediate sentence-level features of the original text and the generated auxiliary text description can be obtained. and A triplet contrastive loss is designed to optimize the sentence-level features of the text (also known as global features) to obtain closely aligned global features of the original text description and the auxiliary video description.

[0058]

[0059] Where B represents the number of text-video pairs in a small batch during training; As an anchor point, is a positive sample, is a negative sample; As an anchor point, is a positive sample, is a negative sample; t corresponds to the given original text; c corresponds to the auxiliary text description generated based on the video.

[0060] Step S200 : Aggregate the word-level features of the target text and the block-level features of the target video respectively to obtain the phrase-level features of the target text and the object-level features of the target video.

[0061] Specifically, this embodiment has an innovative design for the fine-grained features of text and video. For text, some function words may not contain actual semantic information, while some words usually have specific semantic content in the form of phrases; for video, the block-level features of each frame may only correspond to a partial area of ​​a certain object and do not represent specific semantic content. Based on this phenomenon, this embodiment pre-constructs a fine-grained feature aggregation module. After obtaining the word-level features of the text and the block-level features of the video, the fine-grained feature aggregation module is used to aggregate the word-level features of the text and the block-level features of the video to obtain features with actual semantic meaning, namely the phrase-level features of the text and the object-level features of the video. In the semantic matching process of text and video, the phrases in the text description also have semantic associations with the objects in the video, and the alignment of the phrase-level features of the text with the object-level features in the video is also crucial. This embodiment can more effectively achieve fine-grained feature alignment of text and video through the fine-grained feature aggregation module.

[0062] In one implementation, the step of respectively aggregating the word-level features of the target text and the block-level features of the target video to obtain the phrase-level features of the target text and the object-level features of the target video includes:

[0063] Obtaining an auxiliary text description of the target video through the video understanding model, and obtaining word-level features and sentence-level features of the auxiliary text description of the target video through the description encoder;

[0064] Obtaining a common embedding space of the target text and the target video, and generating a unified prompt according to the common embedding space;

[0065] The unified prompt is used as a query vector to interact with the word-level features of the target text; phrase-level features of the target text are obtained based on the word aggregation features obtained after the interaction; wherein, during the training phase, the word aggregation features corresponding to the training data are generated by the interaction of the unified prompt corresponding to the training data, the word-level features of the text, and the sentence-level features of the auxiliary text description;

[0066] The unified prompt is used as a query vector to interact with the block-level features of the target video, or the unified prompt is used as a query vector to interact with the block-level features of the target video and the sentence-level features of the auxiliary text description of the target video; and the object-level features of the target video are obtained based on the block aggregation features obtained after the interaction.

[0067] Furthermore, the step of generating a unified prompt based on the common embedding space is performed by a preset prompt generation module, and the loss function calculation method of the prompt generation module includes:

[0068] Obtaining a plurality of predicted unified prompts generated by the prompt generation module;

[0069] The similarities between each pair of the predicted unified prompts are calculated, and the diversity loss of the prompt generation module is calculated based on all the similarities.

[0070] In summary, this embodiment uses a visual understanding model (or a large visual language model) to obtain auxiliary text descriptions for videos, using them as an additional text modality to enhance the semantic information of the original text. Based on the given original text and the generated auxiliary text descriptions, a coarse-to-fine multi-granularity alignment learning framework is constructed, performing hierarchical alignment based on multi-granularity features such as video-sentence and object-phrase.

[0071] Specifically, although frame-level features represent the global features of each frame in the video, they are still insufficient in terms of fineness. From a semantic point of view, the word-level features of the text cannot be well aligned with the frame-level features of the video. In addition, the block-level features output by the visual encoding are not sufficient to represent semantic features with practical significance. In order to better achieve fine-grained alignment of text and video, this embodiment further aggregates the word-level features of the text and the block-level features of the video, and generates phrase-level features of the text and object-level features of the video under the guidance of a unified prompt generated based on the embedding space shared by the two modalities, and designs a fine-grained alignment module based on these two features. Among them, the unified prompt is designed as a learnable embedding, which is implemented based on the complete fine-tuning of CLIP. In addition, the generated auxiliary text description can also improve the fine-grained aggregation module to remove redundant information in the video and generate semantically rich text features.

[0072] It's important to note that during training, the pairing information between text and video is known. However, during the inference phase, only the paired information between the auxiliary text description and the video is known, and the relevance of the original text and video is unknown. To prevent information leakage, during the inference phase, the auxiliary text description is not used when aggregating word-level features of the text via the fine-grained feature aggregation module.

[0073] For example, this embodiment can use the visual understanding model to obtain auxiliary text descriptions c of the video and use them as additional text modalities to enhance the semantic information of the original text t. The caption encoder and the text encoder share weights to ensure that the obtained text features remain consistent and coherent.

[0074] z c =F t (c);

[0075] Similarly, the feature g corresponding to the [EOS] tag output by the last layer of the CLIP text encoder is c ∈R D As the global features of the generated auxiliary text description, that is, sentence-level features.

[0076] In order to focus on the same semantic information during the text and video feature aggregation process, this embodiment generates a unified prompt from the common space (i.e., the feature space shared by text and video) The aggregation of fine-grained features of text and video is guided by shared learning, where features of different categories in the common space correspond to different semantic information and can generate different unified prompts. At the same time, a diversity loss L is designed. div , so that N q Each unified prompt focuses on different semantic information as much as possible. Intuitively, the similarity between different unified prompts should be 0. Diversity loss L div The calculation formula is as follows:

[0077]

[0078] Among them, u i 、u j represents different unified prompts (i≠j); cos between different prompts represents similarity; diversity loss L div It is used to converge the sum of similarities between different unified prompts so that the similarities between different unified prompts approach 0.

[0079] This embodiment provides two fine-grained feature aggregation methods:

[0080] Method 1: Fine-grained feature aggregation for text: Take the unified prompt u as the query, based on V L The fine-grained aggregation module designed for the Transformer layer enables interaction between unified prompts and word-level features of the text. The features obtained by interacting with the remaining word-level features when the unified prompt is used as a query vector are used as aggregate features of the word-level features, representing the phrase-level semantics of the text. If there are multiple unified prompts, each one will interact with each word-level feature, using its similarity with each word-level feature as a weight to aggregate the focused word-level features.

[0081]

[0082] Fine-grained feature aggregation for videos: Taking the unified prompt u as the query, based on V LThe fine-grained aggregation module designed for the Transformer layer enables interaction between unified cues and video block features. The features obtained by interacting with the remaining block-level features when the unified cues are used as query vectors are used as aggregated features of the block-level features, representing the object-level semantics of the video. If there are multiple unified cues, each one will interact with each block-level feature, using its similarity with each block-level feature as a weight to aggregate the block-level features of interest.

[0083]

[0084] in,

[0085] Method 2: Text-enhanced fine-grained feature aggregation is used. In order to aggregate the fine-grained features of paired text and video with more similar semantic information, the auxiliary text description generated by the video understanding model is used to assist in the aggregation of fine-grained features. Figure 4 As shown, using N L The fine-grained aggregation module, composed of a 10-layer Transformer, obtains phrase-level features of the text based on the word-aggregated features obtained after the interaction of unified prompts, sentence-level features of the generated auxiliary text description, and word-level features of the original text. The object-level features of the video are obtained based on the block-aggregated features obtained after the interaction of unified prompts, sentence-level features of the generated auxiliary text description, and block-level features of the video, thereby achieving better aggregation of fine-grained features. It should be noted that for text, the fine-grained feature aggregation method based on text enhancement is only applicable to the training phase, not the inference phase, to avoid information leakage.

[0086] The fine-grained feature aggregation of text enhancement is specifically reflected in:

[0087]

[0088]

[0089] In one implementation, the unified hint is used as a query vector to interact with the block-level features of the target video, and object-level features of the target video are obtained based on the block-aggregated features obtained after the interaction, including:

[0090] Calculating a similarity score between the sentence-level features of the target text and the frame-level features of the target video, and screening out a number of important frames based on the similarity score;

[0091] The object-level features corresponding to each of the important frames are used as the object-level features of the target video.

[0092] Specifically, considering that the text may only be aligned with some frames in the video, and in order to reduce the amount of calculation in fine-grained similarity calculation, this embodiment pre-designs a similarity perception module (such as Figure 4 (As shown in the figure). This similarity perception module calculates the similarity between sentence-level features of the text and video frame features and selects several important frames of the video based on the similarity. During the subsequent fine-grained alignment, only the fine-grained similarity between the phrase-level features of the text and the object-level features of these important frames is calculated.

[0093] For example, in order to further reduce the amount of calculation, this embodiment selects several important frames from all the obtained frames based on the similarity-aware strategy, and only retains the object-level features of these important frames for calculation, where the importance of the frame is measured by the similarity score between the sentence-level features of the given original text and the frame-level features of the video.

[0094]

[0095] Among them, S represents the similarity vector between the sentence-level features of the text and the frame features of the video, and s represents the cosine similarity.

[0096] Calculate the sentence-level features g of the text t and frame-level features of the video The similarity scores between them are calculated and the frames are sorted based on the similarity scores, and the N frames with the highest similarity are selected. f The object-level features corresponding to the frames (i.e., the Top-k frames in the following formula) are used as fine-grained features of the video for subsequent fine-grained alignment with the phrase-level features of the text.

[0097] Sel_idx=Top-k(S);

[0098]

[0099] in, represents all object-level features of the video v before frame selection, Each video contains F frames, and each frame contains N q object-level features. represents all object-level features of the video v after frame selection, Each video contains N F Frames, each frame contains N q object-level features.

[0100] Step S300: Perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score.

[0101] Specifically, this embodiment implements multi-granular semantic alignment between text and video from coarse to fine. For coarse-grained alignment, after using the attention mechanism in the aforementioned steps to interact the sentence-level features of the text with the frame-level features of the video to obtain global video features, it performs coarse-grained alignment with the sentence-level features of the text to obtain a coarse-grained similarity score. For fine-grained alignment, after aggregating the word-level features of the text and the block-level features of the video in the aforementioned steps to obtain the phrase-level features of the text and the object-level features of the video, it performs fine-grained alignment based on these two features to obtain a fine-grained similarity score.

[0102] In one implementation, fine-grained alignment is performed based on phrase-level features of the target text and object-level features of the target video to obtain a fine-grained similarity score; and coarse-grained alignment is performed based on sentence-level features of the target text and global video features of the target video to obtain a coarse-grained similarity score, including:

[0103] Calculating a similarity matrix based on the phrase-level features of the target text and the object-level features of the target video through matrix multiplication, and aggregating the similarity matrix through a Softmax function to obtain a fine-grained similarity score;

[0104] A coarse-grained similarity score is calculated by matrix multiplication based on the sentence-level features of the target text and the global video features of the target video.

[0105] Specifically, if Figure 5 As shown in the figure, for coarse-grained alignment, matrix multiplication is used to calculate similarity based on the global features of the video and the sentence-level features of the text. This is then quantified into a coarse-grained similarity score to achieve coarse-grained alignment of the text and video. For fine-grained alignment, matrix multiplication is used to calculate the similarity matrix based on the local object-level features of the video and the phrase-level features of the text. Softmax is used to calculate the weighted sum and aggregate the results into a fine-grained similarity score to achieve fine-grained alignment of the text and video. Based on these two similarity scores, the overall similarity between the text and video can be accurately obtained.

[0106] In one implementation, the steps of fine-grained alignment and coarse-grained alignment are performed by a preset multi-grained alignment module; the multi-grained alignment module includes a coarse-grained alignment module and a fine-grained alignment module; the corresponding loss function calculation methods during training of the coarse-grained alignment module and the fine-grained alignment module include:

[0107] Divide the training data in the training dataset into several small batches of data of the same size;

[0108] Selecting a small batch of data, obtaining a coarse-grained similarity matrix corresponding to the small batch of data through the coarse-grained alignment module; calculating the information entropy loss of coarse-grained contrastive learning based on the coarse-grained similarity matrix; wherein the information entropy loss of the coarse-grained contrastive learning is calculated based on constraints in the row and column directions of the coarse-grained similarity matrix;

[0109] A fine-grained similarity matrix corresponding to the small batch data is obtained through the fine-grained alignment module; the information entropy loss of fine-grained contrastive learning is calculated based on the fine-grained similarity matrix; wherein the information entropy loss of the fine-grained contrastive learning is calculated based on the constraints in the row and column directions of the fine-grained similarity matrix.

[0110] For example, for coarse-grained alignment: based on the global feature g of the video v ∈R D And the sentence-level features of the text g t ∈R D , calculate its similarity S through matrix multiplication Co , to achieve coarse-grained alignment of text and video.

[0111] S Co =(g t ) T g v ∈R 1 ;

[0112] During the training process, the data in the dataset is processed in batches, and a small batch of data of size B is processed each time to obtain a similarity matrix R of size B×B Co , using the information entropy loss (InfoNCE Loss) of contrastive learning, the similarity scores of semantically related text videos in the coarse-grained similarity matrix are made higher to optimize the retrieval model.

[0113]

[0114]

[0115]

[0116] Among them, t2v represents the constraint in the row direction, that is, text retrieves video, v2t represents the constraint in the column direction, that is, video retrieves text, They represent the diagonal elements of the coarse-grained similarity matrix, the i-th row and j-th column elements of the coarse-grained similarity matrix (off-diagonal elements of each row), and the j-th row and i-th column elements of the coarse-grained similarity matrix (off-diagonal elements of each column). The optimization goal is to make the similarity on the diagonal of the matrix higher, that is, the similarity between semantically related text and video is higher, thereby achieving alignment between text and video.

[0117] Towards fine-grained alignment: local object-level features from videos and phrase-level features of text Use matrix multiplication to calculate its similarity matrix The weights are calculated by Softmax and then weighted summed to aggregate them into an example-level similarity score, which is the final fine-grained similarity score, to achieve fine-grained alignment of text and video.

[0118]

[0119] Among them, S Fi(i,j) Represents the i-th row and j-th column element of the fine-grained similarity matrix, thereby obtaining a size of N f ×N q The similarity matrix represents the fine-grained similarity score between video v and text t. The similarity matrix is ​​further aggregated to obtain the fine-grained example-level similarity score S″ between video v and text t. Fi .

[0120]

[0121]

[0122] Similarly, during training, given a small batch of data of size B, a fine-grained similarity matrix R of size B×B is obtained through the fine-grained alignment module Fi , using the information entropy loss (InfoNCE Loss) of contrastive learning, the similarity scores of semantically related text videos in the fine-grained similarity matrix are higher to optimize the retrieval model.

[0123]

[0124]

[0125]

[0126] in, They respectively represent the diagonal elements of the fine-grained similarity matrix, the elements of the i-th row and j-th column of the fine-grained similarity matrix (the non-diagonal elements of each row of the fine-grained similarity matrix), and the elements of the j-th row and i-th column of the fine-grained similarity matrix (the non-diagonal elements of each column of the fine-grained similarity matrix).

[0127] Step S400: Determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video.

[0128] In summary, after obtaining the coarse-grained similarity score and the fine-grained similarity score through the coarse-grained alignment module and the fine-grained alignment module in the aforementioned steps, based on these two similarity scores, the overall similarity between the text and the video can be finally obtained, and mutual retrieval between the text and the video can be achieved based on the overall similarity.

[0129] Specifically, in actual text-video retrieval scenarios, the text-video retrieval task falls into two main directions. The first direction is text-video retrieval, which ranks videos in the video library V based on the similarity calculated between a given text query t and a video v∈V. The second direction is video-text retrieval, which ranks texts in the text library T based on the similarity calculated between a given video query v and a text t∈T. The text-video retrieval task aims to learn a similarity function to calculate the similarity score s(t,v) between a given text t and a video v. Therefore, the text-video similarity calculation method provided in this embodiment is well suited for text-video retrieval tasks.

[0130] For example, given a text t and a video v, the coarse-grained alignment module and the fine-grained alignment module are used to obtain the coarse-grained similarity scores S and S, respectively. Co ∈R 1 and fine-grained similarity score S″ Fi ∈R 1 Based on these two similarity scores, we can obtain the example-level similarity S of the text video as a whole. total ∈R 1 .

[0131] S total =S Co +S″ Fi .

[0132] In one implementation, the text encoder, description encoder, video encoder, prompt generation module, fine-grained aggregation module, and multi-granularity alignment module in the aforementioned steps can be integrated into a multi-granularity cross-modal alignment algorithm model. The objective function of this integrated algorithm model consists of the triple loss function of the affinity alignment module, the diversity loss function of the prompt generation module, the coarse-grained contrastive learning loss function of the coarse-grained alignment module, and the fine-grained contrastive learning loss function of the fine-grained alignment module.

[0133] Specifically expressed as:

[0134] L=L Co +α×L Fi +β×L triplet +γ×L div ;

[0135] Among them, α, β and γ represent the learning weights of the fine-grained contrastive learning loss function, the triplet loss function and the diversity loss function, respectively.

[0136] Furthermore, during the algorithm model testing phase, the trained model weights are loaded. Through the two phases of feature extraction and fine-grained feature aggregation, coarse-grained and fine-grained features for text, and coarse-grained and fine-grained features for video, respectively, are obtained. A coarse-grained alignment module and a fine-grained alignment module are used to obtain a coarse-grained similarity matrix and a fine-grained similarity matrix. The coarse-grained similarity matrix reflects the coarse-grained similarity score of the current text and video, while the fine-grained similarity matrix reflects the fine-grained similarity score of the current text and video. The coarse-grained and fine-grained similarity matrices are then added together to obtain the final similarity matrix for the test set. This similarity matrix is ​​a numerical matrix, where the values ​​of the rows (or the depth of the blocks) correspond to the similarity between each text and each video, and the values ​​of the columns (or the depth of the blocks) correspond to the similarity between each video and each text. Based on this similarity matrix, the final similarity score for the text and video is determined, thereby enabling text-video retrieval and ensuring the reliability and accuracy of the model during the inference phase.

[0137] The innovation of the present invention is:

[0138] (1) Based on the most advanced video understanding model, this paper generates corresponding auxiliary text descriptions for videos as a semantic supplement to the original text, expanding the semantic information of the text and better aligning the textual semantic information with the rich semantic content of the video. Furthermore, an affinity comparison module is designed to enhance the semantic capabilities of the auxiliary text descriptions, achieving close semantic alignment between the original and auxiliary text descriptions.

[0139] (2) This paper proposes a text-enhanced multi-granularity cross-modal alignment framework, which includes feature extraction, fine-grained feature aggregation, and multi-granularity cross-modal alignment, to better serve text-video retrieval. Specifically, based on the original text and the generated auxiliary text description, this paper constructs a coarse-to-fine multi-granularity alignment learning framework. It performs hierarchical alignment based on multi-granularity features such as video-sentence and object-phrase, achieving more refined and accurate text-video matching to enhance text-video retrieval performance.

[0140] (3) The present invention utilizes auxiliary text descriptions, and under the guidance of unified prompts generated in the common embedding space, generates phrase-level features of text and object-level features of video through a fine-grained feature aggregation module, and performs frame selection based on the similarity perception module to reduce the amount of fine-grained similarity calculation, thereby achieving more refined and accurate text-video matching.

[0141] Based on the above embodiments, the present invention also provides a text and video similarity calculation system based on multi-granularity alignment, such as Figure 6 As shown, the system includes:

[0142] Feature extraction module 01 is used to obtain coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively. The fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features.

[0143] Feature aggregation module 02, for respectively aggregating the word-level features of the target text and the block-level features of the target video to obtain phrase-level features of the target text and object-level features of the target video;

[0144] The multi-granularity alignment module 03 is configured to perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score;

[0145] Comprehensive analysis module 04 is used to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to implement the retrieval of videos in the video library given a text; or to implement the retrieval of texts in the text library given a video.

[0146] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 7As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected via a system bus. The processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for calculating the similarity of text and video based on multi-granularity alignment is implemented. The display screen of the terminal can be a liquid crystal display or an electronic ink display.

[0147] Those skilled in the art will understand that Figure 7 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0148] In one implementation, the terminal has one or more programs stored in its memory and is configured to be executed by one or more processors. The one or more programs include instructions for performing a method for calculating similarity between text and video based on multi-granularity alignment.

[0149] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0150] In summary, the present invention discloses a method and system for calculating the similarity of text and video based on multi-granularity alignment. By deeply analyzing the hierarchical features contained in text and video, a multi-granularity alignment framework from coarse to fine is designed to achieve more refined and accurate semantic matching. In particular, in terms of fine-grained alignment, by aggregating the word-level features of the text and the block-level features of the video, the phrase-level features of the text and the object-level features of the video are obtained, and based on these two features, the fine-grained alignment of the text and the video is achieved, making the fine-grained semantic alignment more reasonable, thereby effectively solving the problems of missing detailed information and cross-modal semantic gaps in text and video retrieval.

[0151] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A method for calculating similarity between text and video based on multi-granularity alignment, characterized in that: The method comprises: Obtaining coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, including: obtaining word-level features and sentence-level features corresponding to the target text through a text encoder; obtaining frame-level features and block-level features corresponding to each frame of the target video through a visual encoder, and obtaining a frame-level feature sequence based on the frame-level features of each frame; generating a target frame-level feature sequence containing timing information based on the frame-level feature sequence through a temporal encoder; aggregating the target frame-level feature sequence based on the sentence-level features of the target text through a cross-attention mechanism to obtain global video features of the target video; the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features; Aggregating the word-level features of the target text and the block-level features of the target video respectively to obtain phrase-level features of the target text and object-level features of the target video; Performing fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; performing coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score; Determining the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video; The text encoder is trained by a preset affinity alignment module, which is used to: Obtain training data; wherein the training data includes text and video with corresponding relationships, and auxiliary text descriptions generated based on the video by a video understanding model; Inputting the text of the training data into the initial text encoder to obtain sentence-level features of the text; and inputting the auxiliary text description of the training data into the initial description encoder to obtain sentence-level features of the auxiliary text description; Obtaining, through the internal layers of the text encoder and the description encoder, intermediate sentence-level features corresponding to the text and the auxiliary text description, respectively; A triplet contrast loss is calculated based on the two intermediate sentence-level features to train the text encoder; wherein the text encoder shares weights with the description encoder.

2. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 1, characterized in that: The steps of respectively aggregating the word-level features of the target text and the block-level features of the target video to obtain the phrase-level features of the target text and the object-level features of the target video include: Obtaining an auxiliary text description of the target video through the video understanding model, and obtaining word-level features and sentence-level features of the auxiliary text description of the target video through the description encoder; Obtaining a common embedding space of the target text and the target video, and generating a unified prompt according to the common embedding space; The unified prompt is used as a query vector to interact with the word-level features of the target text; phrase-level features of the target text are obtained based on the word aggregation features obtained after the interaction; wherein, during the training phase, the word aggregation features corresponding to the training data are generated by the interaction of the unified prompt corresponding to the training data, the word-level features of the text, and the sentence-level features of the auxiliary text description; The unified prompt is used as a query vector to interact with the block-level features of the target video, or the unified prompt is used as a query vector to interact with the block-level features of the target video and the sentence-level features of the auxiliary text description of the target video; and the object-level features of the target video are obtained based on the block aggregation features obtained after the interaction.

3. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 2, characterized in that: The step of generating a unified prompt based on the common embedding space is performed by a preset prompt generation module, and the loss function calculation method of the prompt generation module includes: Obtaining a plurality of predicted unified prompts generated by the prompt generation module; The similarities between each pair of the predicted unified prompts are calculated, and the diversity loss of the prompt generation module is calculated based on all the similarities.

4. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 2, characterized in that: The unified hint is used as a query vector to interact with the block-level features of the target video, and object-level features of the target video are obtained according to the block-aggregated features obtained after the interaction, including: Calculating a similarity score between the sentence-level features of the target text and the frame-level features of the target video, and screening out a number of important frames based on the similarity score; The object-level features corresponding to each of the important frames are used as the object-level features of the target video.

5. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 1, characterized in that: Performing fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; Coarse-grained alignment is performed based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score, including: Calculating a similarity matrix based on the phrase-level features of the target text and the object-level features of the target video through matrix multiplication, and aggregating the similarity matrix through a Softmax function to obtain a fine-grained similarity score; A coarse-grained similarity score is calculated by matrix multiplication based on the sentence-level features of the target text and the global video features of the target video.

6. The method for calculating similarity between text and video based on multi-granularity alignment according to claim 5, characterized in that: The steps of fine-grained alignment and coarse-grained alignment are performed by a preset multi-grained alignment module; The multi-granularity alignment module includes a coarse-granularity alignment module and a fine-granularity alignment module; The loss function calculation methods corresponding to the coarse-grained alignment module and the fine-grained alignment module during training include: Divide the training data in the training dataset into several small batches of data of the same size; Selecting a small batch of data, obtaining a coarse-grained similarity matrix corresponding to the small batch of data through the coarse-grained alignment module; calculating the information entropy loss of coarse-grained contrastive learning based on the coarse-grained similarity matrix; wherein the information entropy loss of the coarse-grained contrastive learning is calculated based on constraints in the row and column directions of the coarse-grained similarity matrix; A fine-grained similarity matrix corresponding to the small batch data is obtained through the fine-grained alignment module; the information entropy loss of fine-grained contrastive learning is calculated based on the fine-grained similarity matrix; wherein the information entropy loss of the fine-grained contrastive learning is calculated based on the constraints in the row and column directions of the fine-grained similarity matrix.

7. A text and video similarity calculation system based on multi-granularity alignment, characterized in that: The system comprises: A feature extraction module is used to obtain coarse-grained features and fine-grained features corresponding to the target text and target video to be matched, respectively, including: obtaining word-level features and sentence-level features corresponding to the target text through a text encoder; obtaining frame-level features and block-level features corresponding to each frame of the target video through a visual encoder, and obtaining a frame-level feature sequence based on the frame-level features of each frame; generating a target frame-level feature sequence containing timing information based on the frame-level feature sequence through a temporal encoder; aggregating the target frame-level feature sequence based on the sentence-level features of the target text through a cross-attention mechanism to obtain the global video features of the target video; the fine-grained features of the target text are word-level features, and the coarse-grained features are sentence-level features; the fine-grained features of the target video are block-level features of each frame, and the coarse-grained features are global video features; A feature aggregation module, configured to aggregate the word-level features of the target text and the block-level features of the target video to obtain phrase-level features of the target text and object-level features of the target video; A multi-granularity alignment module is configured to perform fine-grained alignment based on the phrase-level features of the target text and the object-level features of the target video to obtain a fine-grained similarity score; and to perform coarse-grained alignment based on the sentence-level features of the target text and the global video features of the target video to obtain a coarse-grained similarity score; a comprehensive analysis module, configured to determine the overall similarity between the target text and the target video based on the fine-grained similarity score and the coarse-grained similarity score; wherein the overall similarity is used to retrieve videos in a video library given a text; or to retrieve texts in a text library given a video; The text encoder is trained by a preset affinity alignment module, which is used to: Obtain training data; wherein the training data includes text and video with corresponding relationships, and auxiliary text descriptions generated based on the video by a video understanding model; Inputting the text of the training data into the initial text encoder to obtain sentence-level features of the text; and inputting the auxiliary text description of the training data into the initial description encoder to obtain sentence-level features of the auxiliary text description; Obtaining, through the internal layers of the text encoder and the description encoder, intermediate sentence-level features corresponding to the text and the auxiliary text description, respectively; A triplet contrast loss is calculated based on the two intermediate sentence-level features to train the text encoder; wherein the text encoder shares weights with the description encoder.

8. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the method for calculating the similarity of text and video based on multi-granularity alignment as described in any one of claims 1 to 6 above.

Images