Text-video retrieval method based on multi-granularity attention

Abstract

The invention discloses a text-video retrieval method based on multi-granularity attention, which comprises the following steps of: firstly, aiming at a given text-video pair, generating disturbance text characteristics through a learnable semantic preserving strategy, and constructing a text with semantic key conflicts as a difficult negative sample to carry out comparative learning so as to strengthen semantic discrimination capability; secondly, short-time actions, medium-time semantics and long-time features are extracted from the video, video content is represented in an omnibearing mode, text features serve as query signals, multi-granularity video features are dynamically weighted and fused through a multi-granularity attention mechanism, and accurate alignment of the text and the video features is achieved; and finally, calculating a similarity score of the query text and the fused video features to retrieve a video matched with the query text. According to the method, multi-granularity video features in a text-video retrieval method are considered, adaptive association of text and video semantics is established, and accurate matching is realized to improve the retrieval effect.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal information retrieval, and in particular to a text-video retrieval method based on multi-granularity attention. BACKGROUND

[0002] With the rapid growth of multimedia content, text-video retrieval technology has shown important application value in many practical application scenarios. Typical applications include: content recommendation and intelligent search in short video platforms, abnormal behavior recognition and event backtracking in intelligent security systems, teaching video positioning and knowledge point retrieval in online education, barrier-free human-computer interaction systems for visually impaired people, and automatic screening of sensitive information in large-scale content review, etc. In these scenarios, users usually express retrieval intentions in natural language form, and the system needs to accurately return video clips or complete videos with high semantic matching from a large number of video libraries.

[0003] However, the video content in real scenarios has high complexity and diversity, not only containing rich visual objects, dynamic actions and temporal evolution process, but also involving multi-role interaction, state transition and causal logic and other deep semantic structures. At the same time, the text query of the user often has problems such as ambiguous expression, lexical ambiguity or brief sentence, which further aggravates the difficulty of semantic alignment. Existing methods are fragile when dealing with samples with similar syntactic structures but with key semantic deviations, and often fail to distinguish the corresponding action time difference, thus matching the wrong but grammatically correct video as the positive sample. This error is due to the lack of challenging negative sample guidance in the model training process, which causes the method to only learn coarse-grained semantic matching, while ignoring fine-grained semantic elements such as action logic, role interaction and state change. Videos have rich temporal structures, including not only local details such as instantaneous actions, but also event structures spanning several seconds or even longer. However, most existing methods only independently encode or simply aggregate uniformly sampled frames, ignoring the importance difference of semantic units at different time granularities. This makes it difficult for existing methods to effectively capture the long-term semantic coherence while preserving the key frame details, resulting in a decline in retrieval performance in complex scenarios. SUMMARY

[0004] In order to overcome the problem that the current text-video retrieval method lacks the ability to distinguish subtle semantic differences and the limitation of single time scale modeling video semantic expression, the application proposes a text-video retrieval method based on multi-granularity attention, which enhances fine-grained discrimination ability by generating text difficult negative samples, and improves the coverage of video semantic by multi-granularity attention.

[0005] The technical solution adopted by the application to solve the technical problems is: A text-video retrieval method based on multi-granularity attention is proposed. First, for a given text-video pair, perturbation text features are generated through a learnable semantic preservation strategy, and texts with semantic key conflicts are constructed as difficult negative samples for comparative learning to enhance semantic discrimination ability. Second, short-term action, mid-term semantic, and long-term features are extracted from the video to comprehensively represent the video content. Using text features as query signals, multi-granularity video features are dynamically weighted and fused through a multi-granularity attention mechanism to achieve accurate alignment between text and video features. Finally, the similarity score between the query text and the fused video features is calculated to retrieve videos that match the query text.

[0006] Furthermore, the method includes the following steps: Step 1: The text-video set is represented as follows ,in Represents the set of elements. A text-video pair, Represents the number of elements in the set. It is a text description. for The corresponding video, among which Indicates video The first in frame, express The number of medium frames; Step 2: Perform data preprocessing on the text-video set. Choose any text-video pair First of all, it Perform uniform sampling and extract The frame is used as a representative visual segment, and the sampled frame is processed using a pre-trained CLIP visual encoder. Each frame of the image is encoded to obtain the video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded The dimension of the feature vector space; for text Semantic encoding was performed using the CLIP text encoder to obtain... Basic text feature vectors Traversing the text-video collection Following this method, the visual feature vector and the basic text feature vector for each text-video pair are obtained, thus yielding the video feature vector set. and the set of basic text feature vectors .

[0007] Furthermore, the method also includes the following steps: Step 3: In the text-video collection Choose one of the text descriptions The part-of-speech (POS) analysis tool Spacy was used to identify key components such as nouns, verbs, and adjectives; the pre-trained language model RoBERTa was used to analyze... Four semantic perturbations are performed to generate... Four types of textual difficult negative samples , , and , among which, exchange The nouns in the two positions get ;replace One of the adjectives is obtained ;replace Get a verb ;replace One of the nouns was obtained Traversing the text-video collection This method yields a set of difficult negative samples from all text descriptions. Step 4: Process the collection using the CLIP text encoder. Encoding all elements yields a set of feature vectors for difficult negative samples in the text. ; Step 5: In the basic text feature vector set Choose any one of the text feature vectors and from the set of feature vectors of difficult negative samples in the text Obtain the corresponding text hard negative sample feature vector. , , and Generate corresponding text feature vectors with random perturbations. , , , and , ; in, Represents cosine similarity. This represents the learnable weight matrix. Represents an exponential function. A random perturbation vector sampled from a standard normal distribution; traversing the set of basic text feature vectors. The perturbation text feature vector set is obtained using this method. and the set of feature vectors of difficult negative samples of perturbed text .

[0008] Furthermore, the method also includes the following steps: Step 6: In the video feature vector set Choose any one of the video feature vectors Generate its short-term features Characteristics of the Middle Time and long-term characteristics

[0009] ; ; ; in, Indicates by window length sliding step size For video feature vectors Perform dimension-wise max pooling, selecting the maximum value of each feature dimension within each window to form a sequence of dimensionality-reduced feature vectors, and then concatenate them in chronological order to output the feature vectors. , The number of sliding windows. Indicates the first The vector formed by max pooling all frame features within a sliding window. , , Indicates the first The vector formed by max pooling all frame features within a sliding window. ; Traverse the set of video feature vectors This method yields a multi-granularity video feature vector set. ; Step 7: In the multi-granularity video feature vector set Choose any set of multi-granularity video feature vectors Generate corresponding enhanced multi-granularity video feature vectors.

[0010] ; in, Indicates calculation and The similarity score for each feature; Indicates from Select the top score from the similarity scores One value, Indicates in The Middle A person The visual feature vector corresponding to the selected effective features; traversing the multi-granularity video feature vector set. This method yields an enhanced multi-granularity video feature vector set. .

[0011] Furthermore, the method also includes the following steps: Step 8: Enhance the multi-granularity video feature vector set Choose any group By combining short-time, medium-time, and long-time features, a comprehensive video feature vector is generated. ; in These are learnable weighted parameters; traversing and enhancing the set of multi-granular video feature vectors. This method yields a comprehensive set of video feature vectors. ; Step 9: Calculate the text-to-video loss. ; in, Indicates the first A perturbation text feature vector With the A comprehensive video feature vector Cosine similarity; Step 10: Calculate the video-to-text loss. ; in, Indicates the first A comprehensive video feature vector With the A perturbation text feature vector cosine similarity, Indicates the first A comprehensive video feature vector With the A perturbation text feature vector The corresponding number A perturbation text difficult negative sample feature vector Cosine similarity; Step 11: Calculate the total loss ; Step 12: Repeat steps 3 through 11 until... The calculation ends when the value is less than the specified minimum loss. Step 13: For the query text, select the video with the highest text-to-video similarity score as the search result.

[0012] The technical concept of this invention is as follows: This invention enhances fine-grained discrimination capability by generating grammatically reasonable but semantically conflicting negative samples through semantic perturbation, and models videos at short, medium and long time scales to achieve hierarchical and accurate cross-modal alignment.

[0013] The beneficial effects of this invention are: it enhances the sensitivity of video representation to fine-grained semantic differences such as action sequence and role relationships, strengthens the multi-scale representation capability of video, and improves the accuracy of text-video retrieval methods. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating a text-video retrieval method based on multi-granularity attention. Detailed Implementation

[0015] The invention will now be further described with reference to the accompanying drawings.

[0016] Reference Figure 1 A text-video retrieval method based on multi-granularity attention includes the following steps: Step 1: The text-video set is represented as follows ,in Represents the set of elements. A text-video pair, Represents the number of elements in the set. It is a text description. for The corresponding video, among which Indicates video The first in frame, express The number of medium frames; Step 2: Perform data preprocessing on the text-video set. Choose any text-video pair First of all, it Perform uniform sampling and extract The frame is used as a representative visual segment, and the sampled frame is processed using a pre-trained CLIP visual encoder. Each frame of the image is encoded to obtain the video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded The dimension of the feature vector space; for text Semantic encoding was performed using the CLIP text encoder to obtain... Basic text feature vectors Traversing the text-video collection Following this method, the visual feature vector and the basic text feature vector for each text-video pair are obtained, thus yielding the video feature vector set. and the set of basic text feature vectors ; Step 3: In the text-video collection Choose one of the text descriptions The part-of-speech (POS) analysis tool Spacy was used to identify key components such as nouns, verbs, and adjectives; the pre-trained language model RoBERTa was used to analyze... Four semantic perturbations are performed to generate... Four types of textual difficult negative samples , , and , among which, exchange The nouns in the two positions get ;replace One of the adjectives is obtained ;replace Get a verb ;replace One of the nouns was obtained Traversing the text-video collection This method yields a set of difficult negative samples from all text descriptions. ; Step 4: Process the collection using CLIP text encoder. Encoding all elements yields a set of feature vectors for difficult negative samples in the text. ; Step 5: In the basic text feature vector set Choose any one of the text feature vectors and from the set of feature vectors of difficult negative samples in the text Obtain the corresponding text hard negative sample feature vector. , , and Generate corresponding text feature vectors with random perturbations. , , , and , ; in, Represents cosine similarity. This represents the learnable weight matrix. Represents an exponential function. A random perturbation vector sampled from a standard normal distribution; traversing the set of basic text feature vectors. The perturbation text feature vector set is obtained using this method. and the set of feature vectors of difficult negative samples of perturbed text ; Step 6: In the video feature vector set Choose any one of the video feature vectors Generate its short-term features Characteristics of the Middle Time and long-term characteristics

[0017] ; ; ; in, Indicates by window length sliding step size For video feature vectors Perform dimension-wise max pooling, selecting the maximum value of each feature dimension within each window to form a sequence of dimensionality-reduced feature vectors, and then concatenate them in chronological order to output the feature vectors. , The number of sliding windows. Indicates the first The vector formed by max pooling all frame features within a sliding window. , , Indicates the first The vector formed by max pooling all frame features within a sliding window. ; Traverse the set of video feature vectors This method yields a multi-granularity video feature vector set. ; Step 7: In the multi-granularity video feature vector set Choose any set of multi-granularity video feature vectors Generate corresponding enhanced multi-granularity video feature vectors.

[0018] ; in, Indicates calculation and The similarity score for each feature; Indicates from Select the top score from the similarity scores One value, Indicates in The Middle A person The visual feature vector corresponding to the selected effective features; traversing the multi-granularity video feature vector set. This method yields an enhanced multi-granularity video feature vector set. ; Step 8: Enhance the multi-granularity video feature vector set Choose any group By combining short-time, medium-time, and long-time features, a comprehensive video feature vector is generated. ; in These are learnable weighted parameters; traversing and enhancing the set of multi-granular video feature vectors. This method yields a comprehensive set of video feature vectors. ; Step 9: Calculate the text-to-video loss. ; in, Indicates the first A perturbation text feature vector With the A comprehensive video feature vector Cosine similarity; Step 10: Calculate the video-to-text loss. ; in, Indicates the first A comprehensive video feature vector With the A perturbation text feature vector cosine similarity, Indicates the first A comprehensive video feature vector With the A perturbation text feature vector The corresponding number A perturbation text difficult negative sample feature vector Cosine similarity; Step 11: Calculate the total loss ; Step 12: Repeat steps 3 through 11 until... The calculation ends when the value is less than the specified minimum loss. Step 13: For the query text, select the video with the highest text-to-video similarity score as the search result.

[0019] In this embodiment, for the text-video set, the CLIP text encoder and video encoder are first used to extract feature vectors of the text and video. For the query text "a man in a blue shirt talking about a dog", the corresponding set of difficult negative samples {"a shirt in a blueman talking about a dog", "a man in a striped shirt talking about a dog", "a man in a biue shirt complaining about a dog", "a woman in a blue shirt talking about a dog"} is generated using the POS analysis tools Spacy and RoBERTa, and the feature vectors of the difficult negative samples are extracted using the CLIP text encoder. A perturbed text feature vector is generated through a learnable perturbation mechanism. Guided by the text feature vectors, the video is subjected to multi-granular pooling and based on... Weighted fusion of similar frames is performed, and then a comprehensive video feature vector is synthesized through learnable weights. The similarity score between the perturbed text feature vector and the comprehensive video feature vector is calculated. A text hard negative sample similarity term is introduced to enhance fine-grained discrimination capability. The InfoNCE loss function is used to calculate the bidirectional alignment loss between text-to-video and video-to-text. and By querying the similarity scores between text and video, the system selects the video with the highest similarity score from the video collection, and the retrieved result is the video that correctly corresponds to the query text.

[0020] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A text-video retrieval method based on multi-granularity attention, characterized in that, First, for a given text-video pair, perturbed text features are generated through a learnable semantic preservation strategy, and a text with semantic key conflicts is constructed as a difficult negative sample to carry out contrastive learning, so as to strengthen the semantic discrimination ability; secondly, short-time actions, medium-time semantics and long-time features are extracted from the video to comprehensively represent the video content, and the text features are taken as a query signal to dynamically weight and fuse multi-granularity video features through a multi-granularity attention mechanism, so as to realize accurate alignment of the text and the video features; Finally, the similarity score of the query text and the fused video features is calculated to retrieve the video matched with the query text.

2. The text-video retrieval method based on multi-granularity attention of claim 1, wherein, The method comprises the following steps: Step one, text-video collection is represented as where represents the th text-video pair in the collection, denotes the number of collection elements, is the text description, is the corresponding video, where denotes the th frame in the video , denotes the number of frames in ; Step 2: Perform data preprocessing on the text-video set. Choose any text-video pair First of all, it Perform uniform sampling and extract The frame is used as a representative visual segment, and the sampled frame is processed using a pre-trained CLIP visual encoder. Each frame of the image is encoded to obtain the video. Frame-level visual feature vector sequence ,in Indicates the first The visual feature vector generated after the frame is encoded The dimension of the feature vector space; for text Semantic encoding was performed using the CLIP text encoder to obtain... Basic text feature vectors Traversing the text-video collection Following this method, the visual feature vector and the basic text feature vector for each text-video pair are obtained, thus yielding the video feature vector set. and the set of basic text feature vectors . 3.The text-video retrieval method based on multi-granularity attention of claim 2, wherein, The method further comprises the following steps: Step three, in the text-video set optionally one text description , using the part of speech POS analysis tool Spacy to identify its key components such as nouns, verbs, adjectives, etc.; using the pre-trained language model RoBERTa to perform four kinds of semantic perturbation, respectively generating four categories of text difficult negative samples , , and , wherein, the nouns in are exchanged to obtain ; the adjective in is replaced to obtain ; the verb in is replaced to obtain ; the noun in is replaced to obtain ; traverse all text descriptions in the text-video set , and obtain the text difficult negative sample set according to this method; Step four, encode the set of all elements through the CLIP text encoder to obtain a set of text difficult negative sample feature vectors ;​ Step five, in the basic text feature vector set Optional one text feature vector , and from the text difficult negative sample feature vector set Corresponding text difficult negative sample feature vector , , And , generate corresponding random perturbation text feature vector , , , And , traverse the basic text feature vector set , according to this method to get perturbation text feature vector set And perturbation text difficult negative sample feature vector set .

4. The text-video retrieval method based on multi-granularity attention of claim 3, wherein, The method further comprises the following steps: Step six, generating short-term features , medium-term features and long-term features of each video feature vector in the video feature vector set , , is the number of sliding windows, is the vector composed of the maximum pooling of all frame features in the i-th sliding window, , , , is the vector composed of the maximum pooling of all frame features in the i-th sliding window, ; traversing the video feature vector set , obtaining a multi-granularity video feature vector set according to the method, ; Step seven, generating a corresponding enhanced multi-granularity video feature vector set from the multi-granularity video feature vector set optionally selected from the multi-granularity video feature vector set , generating a corresponding enhanced multi-granularity video feature vector , traversing the multi-granularity video feature vector set , obtaining an enhanced multi-granularity video feature vector set according to the method .

5. The text-video retrieval method based on multi-granularity attention of claim 4, wherein, The method further comprises the following steps: Step eight, traversing the enhanced multi-granularity video feature vector set Optionally, a set of , generating a comprehensive video feature vector by integrating the short-term, medium-term and long-term features; traversing the enhanced multi-granularity video feature vector set , obtaining a set of comprehensive video feature vectors according to the method ; The method further comprises the following steps: Step nine, calculating the text-to-video loss; Step ten, calculating the video-to-text loss; Step twelve, repeat the execution of step three to step eleven, when less than the specified minimum loss value, end the calculation; Step eleven, calculating the total loss; Step thirteen, for a query text, selecting a video with the highest text-to-video similarity score as a retrieval result.

Images