A video retrieval method based on attention segment hinting

Abstract

The application discloses a video retrieval method based on attention fragment prompt. The method comprises the following steps: visual information is extracted from a video, and then corresponding global features are calculated, and text features are extracted based on a query text; the visual information is converted by a time converter, so that each frame of feature has context information; similar video segments in the video are found based on the text features, and the most similar video segment information to the query text is obtained as the key features of the vision; the key features and the global features are weighted and summed to obtain the final visual features; the similarity between the query text and the video is calculated according to the final visual features, and then the target video meeting the similarity requirement is retrieved. The application gives more weight to the most relevant picture while retaining the global information, which is beneficial to accurately retrieving the target video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video retrieval, more particularly, to a video retrieval method based on attention segment prompt. BACKGROUND

[0002] With the rapid development of multimedia technology, the data scale of Internet video has exploded in the past few years, and large-scale visual text pre-training models have promoted the development of video text retrieval. Therefore, for a given text query, how to accurately find the most relevant video from a huge database has become one of the most concerned tasks in the multimedia field.

[0003] In the prior art, different methods are proposed to apply the cross-modal information learned by CLIP (Contrastive Language-Image Pre-training Model) to video retrieval. The video is regarded as a set of continuous pictures, and the temporal relationship between these pictures is learned, so as to extend CLIP in the time dimension. However, the existing methods ignore a key problem in video text retrieval. Since the original video may contain information of multiple scene segments, different annotators will give different text descriptions for this video, and these texts often only describe part of the information of the video. Therefore, it is difficult for the existing methods to match the text information with the rich video information, and it is impossible to locate which segment of the video the text description is.

[0004] In summary, the existing video text retrieval scheme ignores a key problem in video text retrieval (i.e. the text is usually only a small segment of the corresponding video), and usually integrates the entire video without distinction, which inevitably reduces the final retrieval effect, resulting in that it can only be applied to videos with single content. SUMMARY

[0005] The purpose of the present application is to overcome the defects of the prior art, and to provide a video retrieval method based on attention segment prompt, which comprises the following steps:

[0006] The visual information of the video is extracted, and the corresponding global feature is calculated, and the text feature is extracted based on the query text;

[0007] The visual information is converted by a time converter, so that each frame of feature has context information;

[0008] Based on the text feature, similar video segments in the video are found, and the video segment information most similar to the query text is obtained as the key visual feature;

[0009] The key feature and the global feature are weighted and summed to obtain the final visual feature;

[0010] The similarity between the query text and the video is calculated based on the final visual features, and then the target video that meets the similarity requirements is retrieved.

[0011] Compared with existing technologies, the advantages of this invention lie in its design and solution based on the inherent characteristics of video text retrieval. It directly uses text to query the most relevant video frames, preserving global information while assigning more weight to potentially most relevant scenes, thereby improving retrieval accuracy and efficiency. Furthermore, this invention demonstrates good performance even on small datasets and can be easily transferred and integrated into other model frameworks to further improve retrieval performance. Moreover, it is plug-and-play, making it very simple and convenient.

[0012] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0014] Figure 1 This is a flowchart of a video retrieval method based on attention segment cues according to an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of a video retrieval framework based on attention segment cues according to an embodiment of the present invention;

[0016] Figure 3 This is a diagram illustrating the effect of migrating the attention segment prompting module to an existing model according to an embodiment of the present invention. Detailed Implementation

[0017] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0018] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0019] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0020] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0022] This invention proposes a concise Attention Segment Cueing (ASP) framework that dynamically leverages text-related segments from videos to facilitate retrieval. In short, the Attention Segment Cueing framework comprises two key components: segment cues and video aggregation. For a given set of text and video, segment cues cleverly construct text-driven visual cues, dynamically acquiring video segments relevant to the text query. Video aggregation captures global information from the video. Through the fusion of segment cues and video aggregation, the model framework of this invention can effectively learn a robust, text-related visual representation.

[0023] Specifically, in combination Figure 1 and Figure 2 As shown, the provided video retrieval method based on attention fragment cues includes the following steps:

[0024] Step S110: Extract the visual features of the video and the text features of the query text.

[0025] To construct a video retrieval model framework based on attention fragment cues, see [link / reference]. Figure 2 As shown, the framework is illustrated in the form of functional modules. First, it dynamically searches for the most similar video segments based on the query text, obtaining key features of the video related to the query text. Then, it globally integrates the video information to obtain global video features. Finally, it weights and sums the obtained key features and global features to obtain the final visual features (or visual information), and then obtains the video retrieval results based on the final visual features. The framework provided by this invention is more efficient than existing technologies because many original videos contain information from multiple scenes, while query text often only describes a portion of the video, making it difficult to match text information with rich video features. This invention, however, can dynamically find the video segments described by the text, significantly improving video retrieval efficiency.

[0026] Still combined Figure 2Module 1 is used to extract textual and visual features. For example, the video is downsampled to convert it into an image sequence. Then, a Visual Feature Extractor (VIT) is used to extract visual features (or visual information), while a Text Feature Extractor (BERT) is used to extract textual features. The text feature extractor can use BERT or other language models. In one embodiment, VIT and BERT are initialized using weights obtained after CLIP pre-training. The CLIP model utilizes a pre-trained image-text matching task to learn cross-modal information of visual text from large-scale image-text data through self-supervised and contrastive learning. CLIP can be transferred zero-shot to other downstream tasks, and its framework can also serve as the basis for other visual-text tasks to achieve better performance.

[0027] Step S120: Extract key features of the video related to the query text, and fuse the key features with the global visual features to obtain the final visual features.

[0028] After extracting visual and textual features, a novel attention-based cueing module (Module 2) is proposed. Specifically, visual information is processed through a temporal transformer module, ensuring that the features of each frame have contextual information. Then, based on the query text, a similarity comparison is performed with the features of each frame of the video, dynamically acquiring the video segment information most similar to the query text as the key visual features. After obtaining the key text-related features, the visual information extracted by the visual feature extractor (VIT) is averaged to obtain the global visual features. Finally, the key features and the global features are weighted and summed to obtain the final visual features.

[0029] This invention improves computational efficiency and accuracy of feature extraction by extracting key visual features and focusing attention on important information that is highly relevant to the query text, while reducing the number of parameters.

[0030] In one embodiment, key features and global features can be fused using linear weighting or exponential weighting. For example, by continuous experimentation, the optimal hyperparameters are selected to balance the importance of key features (segment level) and global features (video level). Furthermore, for different datasets, the optimal hyperparameters are explored according to the characteristics of each dataset, ultimately achieving superior performance on various types of datasets.

[0031] It should be understood that different visual feature extractors or text feature encoders, or new model frameworks, can also be used. The core of this invention is a text-based cross-modal interaction paradigm. Without departing from the spirit of this invention, those skilled in the art can also choose different interaction functions, more complex interaction modules, or different feature fusion methods. Furthermore, the extraction order of the aforementioned key features and global features can be processed in parallel or sequentially, and this invention does not limit this.

[0032] Step S130: Based on the final visual features, calculate the similarity between the video and the query text to obtain the search results.

[0033] After performing the video feature fusion described above to obtain the final visual features, cross-modal similarity calculation is further performed to output the video retrieval results (see [link]). Figure 2 Module 3).

[0034] In one embodiment, cosine similarity is used as the evaluation criterion for video and text similarity to calculate the similarity between the query text and each video. During model training, a loss function is calculated based on the trained similarity matrix to learn the model. During actual retrieval, one or more of the most similar videos are selected as the retrieved target videos for output based on their similarity scores.

[0035] The loss function of this invention considers the similarity between the query text and the video. This design allows for the direct selection of the most relevant video frames when selectively choosing video frames. In one embodiment, the loss function is set as follows:

[0036]

[0037]

[0038]

[0039] Among them, T (i) Z represents the text features of the i-th text in this batch. (i,j) Let represent the visual features generated by the i-th video in each batch through the fragment prompt of the j-th text; <,> represent the cosine similarity between two features, and M represents the number of videos.

[0040] Using this invention, for a given text description, one or more videos most relevant to the text can be effectively found from a massive video library. This is because, in reality, video clips may include content from more than one frame, including camera cuts, multiple consecutive activities of the target character, etc. For example, in an animated clip, two people are drinking and talking at a restaurant table, and then they walk out of the restaurant talking as they walk. For this video, 20 different annotators watched the video and gave their own annotations. A significant number of them only described part of the video, such as "two animated characters at the table" or "two animated characters walking and talking." Based on these considerations, this invention proposes a targeted approach for video segment recommendation. For example, taking each video frame as the center, a self-attention mechanism is used to integrate information from the preceding and following time series as candidate recommended segments. Then, the cosine similarity between the text representation to be queried and these candidate recommended segments is calculated, and the segment with the highest similarity is directly selected as the recommended segment for local information. At the same time, overall information is also taken into account, using the average pooling of all frame representations as global information. The resulting video representation can dynamically calculate the relevance score to the video for specific text inputs, find the part of the video that is most relevant to the text query, and thus meet the user's personalized query needs.

[0041] To further verify the effectiveness of the present invention, experiments were conducted, and the effectiveness of the present invention was verified from different perspectives such as retrieval accuracy, robustness, and portability.

[0042] 1) Accuracy. Existing technologies integrate video and text information into a single feature space through cross-modal models. In visual features, these methods treat the video as multiple consecutive images and directly perform averaging and summation. This invention, however, dynamically searches for the most similar video segments based on the query text, obtaining key features of the video related to the query text, thereby improving the accuracy of video text retrieval. See Table 1, where our ASP represents the framework of this invention, and the others are existing models.

[0043] 2) Robustness. Existing technologies exhibit poor robustness with limited training data. This invention, however, enhances retrieval robustness by dynamically aligning query text with video clips to obtain the most relevant segments. It also demonstrates good experimental results even with a small training set. See Table 2, which illustrates a comparison of experimental results with CLIP4Clip for training data sizes of 30, 300, and 3000.

[0044] 3) Portability. Existing models are complex and have poor transferability. This application can be easily transferred to existing methods. For example... Figure 3As shown, the attention fragment prompting module is migrated to CLIP2Video, and the experimental results of CLIP2Video, CLIP2Video + attention fragment prompting module and ASP are compared on the MSR-VTT dataset.

[0045] Table 1: Comparison of experimental results with existing video retrieval methods on the MSV-RTT dataset

[0046]

[0047] Table 2: Comparison of CLIP4Clip experimental results with different training data sizes

[0048]

[0049] Extensive experiments on multiple benchmarks demonstrate that the performance of this invention significantly outperforms existing video retrieval methods. For example, on the DiDeMo dataset, results of 49.5R@1 and 13.8MnR are obtained, representing an improvement of over 5% in the R@1 evaluation criterion compared to the recently proposed CLIP4Clip, QB-Norm, and CAMoE.

[0050] In summary, compared with the prior art, the present invention has at least the following technical effects:

[0051] 1) The designed plug-and-play attention-segment prompting module can dynamically obtain relevant video segments based on the query text, enabling different query texts to effectively match different relevant content in the video, thus enhancing the retrieval effect.

[0052] 2) Given a video and human annotations, since the video may contain many frames, while the human annotations only correspond to a small portion, using global information on an average basis will ignore key points. This invention addresses this problem by leveraging the inherent characteristics of video text retrieval, directly using text to query the most relevant video frames. While preserving global information, it assigns more weight to potentially most relevant frames, thus better aiding the retrieval. Furthermore, this invention is very simple and efficient, achieving performance improvements without consuming large-scale computing resources.

[0053] 3) As is well known, data collection is often the most time-consuming and labor-intensive aspect of deep learning tasks. Relying on manually labeled data can also infringe on personal privacy. In many practical situations, due to specific limitations, it is difficult to obtain a large amount of data for training. This invention demonstrates its advantages even on small datasets and exhibits strong transferability, allowing the ASP module to be ported to other retrieval models to further improve retrieval results. For example, conventional models include CLIP and CLIP4Clip, which is an improvement for video. This invention builds upon these models with its own contributions, achieving retrieval performance significantly superior to the CLIP4Clip baseline.

[0054] This invention can be applied to electronic devices, servers, or the cloud to retrieve one or more target videos based on query text. The electronic device can be a terminal device or a server. Terminal devices include any terminal device such as mobile phones, tablets, personal digital assistants (PDAs), point-of-sale (POS) terminals, and smart wearable devices (smartwatches, virtual reality glasses, virtual reality headsets, etc.). Servers include, but are not limited to, application servers or web servers, and can be standalone servers, cluster servers, or cloud servers.

[0055] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0056] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0057] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0058] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0059] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0060] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0061] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0062] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0063] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A video retrieval method based on attention segment cues, comprising the following steps: Visual information is extracted from the video, then the corresponding global features are calculated, and text features are extracted based on the query text. The visual information is transformed using a time converter so that the features of each frame have contextual information. Based on the text features, find similar video segments in the video and obtain the video segment information that is most similar to the query text as the key visual feature; The key features and the global features are weighted and summed to obtain the final visual features; The similarity between the query text and the video is calculated based on the final visual features, and then the target video that meets the similarity requirements is retrieved. The key features are obtained according to the following steps: The text features are compared with the features of each frame of the video to dynamically obtain the video segment information that is most similar to the query text, which is used as the key visual feature. The global features are obtained according to the following steps: The video is downsampled by average to convert it into a sequence of images; The image sequence is input into the Visual Feature Extractor (VIT) to extract visual information; The extracted visual information is averaged to obtain the global visual features.

2. The method according to claim 1, characterized in that, In obtaining the key features, cosine similarity is used as the evaluation criterion for video and text similarity to calculate the similarity between the query text and each video segment.

3. The method according to claim 1, characterized in that, The text features are extracted using the BERT model, and the visual feature extractor (VIT) and the BERT model are initialized using the weights pre-trained with CLIP.

4. The method according to claim 1, characterized in that, The similarity between the query text and the video is calculated based on the final visual features, and then target videos that meet the similarity requirements are retrieved, including: Centered on each video frame, a self-attention mechanism is used to integrate information from the preceding and following time series as candidate recommended segments; The cosine similarity between the text features to be queried and these candidate recommended segments is calculated, and the segment with the highest similarity is directly selected as the recommended segment for local information; at the same time, the average pooling of all frames is used as the global feature. The system dynamically calculates the relevance of the text features to be queried to each video, and finds the video most relevant to the text query.

5. The method according to claim 1, characterized in that, The similarity between the query text and the video is calculated based on the final visual features using a trained cross-modal similarity calculation model.

6. The method according to claim 5, characterized in that, The loss function for training the cross-modal similarity calculation model is calculated based on the similarity matrix between each video segment and text features.

7. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

Images