A video retrieval generation method and device based on sparse representation and re-ranking

Abstract

The present invention discloses a video retrieval generation method and device based on sparse representation and re-ranking, and relates to the field of cross-modal video retrieval technology. The method includes: obtaining a query text, a candidate video sequence, and a video retrieval generation model; obtaining a query text dense representation and a candidate video sequence dense representation based on an encoder module, the query text, and the candidate video sequence; generating a text sparse representation based on the query text dense representation and a candidate video sparse representation based on the candidate video sequence dense representation through a sparse representation module, performing inverted index screening on the candidate video sequence to obtain some candidate videos; determining a global score for each video in some candidate videos through a cross-attention module, and ranking some candidate videos; generating a target text corresponding to the query text through a generation module based on the ranked candidate videos and the query text. The present invention improves retrieval efficiency through sparse representation and re-ranking.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal video retrieval, and in particular to a video retrieval generation method and device based on sparse representation and reordering. Background Art

[0002] Large language models have demonstrated broad applicability across multiple domains. As data volumes increase, they are able to better understand and generate natural language text that is similar to that of humans. However, relying solely on large language models to handle all tasks can encounter bottlenecks, such as delayed knowledge updates and insufficient accuracy in specific domains. To overcome these challenges, researchers have developed Retrieval-Augmented Generation (RAG). RAG combines the strengths of information retrieval and text generation. By incorporating external knowledge sources during task execution, it dynamically acquires relevant information during text generation. This approach not only improves the accuracy of generated content but also ensures the timeliness of the knowledge. Current retrieval-augmented generation (RAG) techniques primarily rely on text or static images, improving generation quality by injecting search results from external knowledge bases into large language models (LLMs). Currently, multimodal RAG systems primarily utilize data sources such as text, images, and tables. However, video, as a medium rich in temporal dynamic information and multimodal semantics, has yet to be fully integrated. In addition, in the current big data environment, with the rapid rise of short videos and self-media, the growth rate of video data is unprecedented. How to quickly and accurately retrieve videos related to text query descriptions from massive videos is an important technical challenge. The accuracy of video retrieval affects the performance of the RAG system.

[0003] Existing RAG systems typically use a two-stage pipeline: Stage 1: Use an encoder (such as CLIP) to map multimodal information into fixed-length vectors, and then retrieve relevant materials through approximate nearest neighbor search (ANN); Stage 2: Directly concatenate the top-K search results into the LLM input for generation. However, this multimodal information only includes text, images, tables, etc., and rarely takes video information into account. To directly migrate existing RAG methods to the video domain, it is necessary to design a dedicated video retrieval method for video RAG. However, existing video retrieval modules often rely on full comparison of high-dimensional features in large-scale video libraries, which has the following shortcomings:

[0004] Current video retrieval techniques generally employ approaches based on multimodal pre-trained models (such as CLIP). These methods encode videos and text into high-dimensional vectors, thereby achieving similarity matching within the vector space. While these approaches offer significant advantages in cross-modal semantic understanding, they suffer from significant time overhead in practical applications. Specifically, they: 1) High computational overhead of high-dimensional vectors: Large-scale video retrieval requires computing similarities between massive amounts of videos and input text. For each video, the video and text vectors provided by the CLIP model are typically high-dimensional (e.g., 512 or higher). This necessitates large-scale dot multiplications of these vectors for each search, significantly increasing computational overhead. 2) Full-scale comparisons lead to search latency: Existing methods often compute similarity based on the entire video set, requiring each query to traverse the entire video collection in the database. For video repositories of millions or more, computational latency and data I / O costs increase significantly, making it difficult to meet real-time response requirements. 3) Inadequate processing of frame-level details: The dynamic frame sequence information of a video may be simplified during compression into a single vector, resulting in the loss of fine-grained temporal information. Even if some models consider frame-level information, this will incur additional storage and online computing overhead. Summary of the Invention

[0005] To address the existing technical issues of high computational overhead for high-dimensional vectors, search delays caused by full comparison, and insufficient processing of frame-level details, the present invention provides a video retrieval generation method and apparatus based on sparse representation and reordering. The technical solution is as follows:

[0006] In one aspect, a video retrieval generation method based on sparse representation and reordering is provided. The method is implemented by a video retrieval generation device based on sparse representation and reordering, and the method includes:

[0007] S1. Obtain query text, candidate video sequences, and a trained video retrieval generation model, wherein the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module, and a generation module;

[0008] S2. Encode the query text and the candidate video sequence into high-dimensional vector representations according to the encoder module to obtain dense representations of the query text and candidate video sequence;

[0009] S3. Generate a text sparse representation based on the query text dense representation through the sparse representation module, generate a candidate video sparse representation based on the candidate video sequence dense representation, and perform inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos;

[0010] S4. Determine a global score for each of the candidate videos based on the query text dense representation and the video dense representation of the candidate videos through a cross-attention module, and sort the candidate videos according to the global score.

[0011] S5. Generate a target text corresponding to the query text according to the sorted candidate videos and the query text through a generation module.

[0012] On the other hand, a video retrieval generation device based on sparse representation and reordering is provided, which is applied to a video retrieval generation method based on sparse representation and reordering, and includes:

[0013] An acquisition unit, configured to acquire a query text, a candidate video sequence, and a trained video retrieval generation model, wherein the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module, and a generation module;

[0014] An encoding unit, configured to encode the query text and the candidate video sequence into high-dimensional vector representations according to the encoder module, to obtain a dense representation of the query text and a dense representation of the candidate video sequence;

[0015] An inverted index unit is used to generate a text sparse representation based on the query text dense representation through a sparse representation module, generate a candidate video sparse representation based on the candidate video sequence dense representation, and perform inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos;

[0016] a ranking unit, configured to determine a global score of each video in the portion of candidate videos based on a dense representation of the query text and dense representations of the portion of candidate videos through a cross-attention module, and to rank the portion of candidate videos according to the global score;

[0017] The generating unit is configured to generate a target text corresponding to the query text according to the sorted candidate videos and the query text through a generating module.

[0018] On the other hand, a video retrieval generation device based on sparse representation and reordering is provided, and the video retrieval generation device based on sparse representation and reordering includes: a processor; a memory, on which computer-readable instructions are stored, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned video retrieval generation methods based on sparse representation and reordering is implemented.

[0019] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned video retrieval generation methods based on sparse representation and reordering.

[0020] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0021] In an embodiment of the present invention, a query text, a candidate video sequence and a trained video retrieval generation model are obtained, and the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module and a generation module; according to the encoder module, the query text and the candidate video sequence are respectively encoded into high-dimensional vector representations to obtain a query text dense representation and a candidate video sequence dense representation; through the sparse representation module, a text sparse representation is generated according to the query text dense representation, and a candidate video sparse representation is generated according to the candidate video sequence dense representation; according to the text sparse representation and the candidate video sparse representation, the candidate video sequence is subjected to inverted index screening to obtain some candidate videos; through the cross-attention module, according to the query text dense representation and the video dense representation of some candidate videos, the global score of each video in some candidate videos is determined, and some candidate videos are sorted according to the global score; through the generation module, according to the sorted candidate videos and the query text, a target text corresponding to the query text is generated. By adopting the present invention, the entire retrieval framework can take into account both efficiency and performance through sparse representation and reordering. The RAG system of external video information and the use of sparse modules and cross-attention modules for rough retrieval and refined reordering provide the RAG system with low-latency, high-precision video retrieval results, allowing the RAG system to effectively utilize external video information to help generate, achieving a balance between efficiency and performance and improving the efficiency of video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of a video retrieval generation method based on sparse representation and reordering provided by an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of a process for establishing and retrieving a sparse index and an inverted index provided by an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of a process for rapid video retrieval enhancement generation provided by an embodiment of the present invention;

[0026] Figure 4This is a block diagram of a video retrieval generation device based on sparse representation and reordering provided by an embodiment of the present invention;

[0027] Figure 5 This is a structural diagram of a video retrieval generation device based on sparse representation and reordering provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0029] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0030] In the embodiments of the present invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same. The terms "of," "corresponding," and "corresponding" may sometimes be used interchangeably. It should be noted that, when the distinction is not emphasized, the meanings they convey are the same.

[0031] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0032] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0033] The embodiment of the present invention provides a video retrieval generation method based on sparse representation and reordering, which can be implemented by a video retrieval generation device based on sparse representation and reordering, and the video retrieval generation device based on sparse representation and reordering can be a terminal or a server. Figure 1 The flowchart of the video retrieval generation method based on sparse representation and re-ranking is shown. The processing flow of the method may include the following steps:

[0034] S1. Obtain query text, candidate video sequences and a trained video retrieval generation model. The video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module and a generation module.

[0035] S2. According to the encoder module, the query text and the candidate video sequence are respectively encoded into high-dimensional vector representations to obtain dense representations of the query text and the candidate video sequence.

[0036] Optionally, the encoder module includes a text encoder and a video encoder, the text encoder adopts the TransformerEncoder architecture, and the video encoder adopts the Vision Transformer architecture.

[0037] In a feasible implementation, a text encoder and a video encoder are used to encode the query text and the candidate video sequence into high-dimensional vector representations, respectively, to provide basic features for subsequent sparsification and attention calculations.

[0038] The specific operation steps of S2 may include the following steps S21-S22:

[0039] S21. Perform word segmentation and embedding on the query text to obtain word vectors, which are then input into the Transformer Encoder layer to obtain a dense representation of the query text.

[0040] S22. Extract multiple key frames from the candidate video sequence by sampling to obtain an input frame set, segment each frame image in the input frame set, obtain an embedded representation based on the segmented image blocks, and input the embedded representation into a video encoder to obtain a dense representation of the candidate video sequence.

[0041] In one feasible implementation, the text encoder adopts the Transformer Encoder architecture and inputs the query text sequence T={w1,w2,…,wN}, where N represents the number of words in the input text. The query text is first segmented and embedded into a word vector {x1,x2,…,x N}∈R N×d , and then fed into the Transformer Encoder layer as input. The output is the context vector representation of each word (i.e. last_hidden_states): H t =Transformer Encoder(x1,x2,…,x N )∈R N×d In order to generate the global feature representation of the text, a special CLS tag [CLS] is added at the beginning of the sequence, and its corresponding vector h [CLS] ∈R d Used as a global feature representation of text: z t =h [CLS] , where z t ∈R dis the global representation vector of the text. After the text encoder, a text query obtains a total of n=N+1 dense representations.

[0042] The video encoder is based on the Vision Transformer (ViT) architecture to encode the candidate video sequence. Since ViT itself is designed to process single-frame images, in order to adapt to the characteristics of the video, the candidate video is sampled to extract M key frames to obtain the input frame set: V={f1,f2,…,f M}, where M is the number of extracted key frames and the size of each frame is H×W×C. Each frame image f i It is divided into patches of fixed size, and each patch is mapped to a 1D token embedding by linear projection: pij =Linear(Flatten(pij)), where pij is the jth patch of the i-th frame, x pij ∈R d Before each frame of image is input to Transformers, a learnable CLS token embedding is concatenated in front of it. The output vector h of CLS token is [CLS]i v Represents the global features of the frame. Finally, after passing through the encoder, the video is encoded into M high-dimensional feature vectors: H v ={h [CLS]1 v , h [CLS]2 v ,…, h [CLS]M v}∈R M×d .like Figure 2 The process of converting text and video into sparse coding is given, and how to perform fast coarse-grained video retrieval through inverted indexing technology is demonstrated.

[0043] S3. Generate a text sparse representation based on the query text dense representation through the sparse representation module, generate a candidate video sparse representation based on the candidate video sequence dense representation, and perform inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos.

[0044] Optionally, the specific operation steps of S3 may include the following steps S31-S33:

[0045] S31. Through the sparse representation module, the dense representation of the query text is projected onto the probability distribution of the full vocabulary to obtain the vocabulary distribution corresponding to the word. Through the maximum pooling operation, a preset value of points are selected in the vocabulary distribution corresponding to the word to obtain the sparse representation of the text.

[0046] S32. Through the sparse representation module, the dense representation of the candidate video sequence is projected onto the probability distribution of the full vocabulary to obtain the vocabulary distribution corresponding to the video frame. Through the maximum pooling operation, a preset value of points are selected from the vocabulary distribution corresponding to the video frame to obtain the sparse representation of the candidate video.

[0047] S33. Calculate preliminary scores of the candidate videos based on the formaldehyde similarity between the text sparse representation and the candidate video sparse representation, sort the multiple preliminary scores from large to small, select the candidate videos corresponding to the preliminary scores in the top N orders, and determine them as some candidate videos.

[0048] In a feasible implementation, the sparse representation module is designed to convert the high-dimensional dense representation output by the encoder into a sparse representation, so as to facilitate subsequent query and establishment of inverted index. Assume that the output of the text encoder is H t ∈R N ×d , the output of the video encoder is H v ∈R M×d The sparse module generates sparse representations by the following steps: First, t and H v Each vector in is projected onto the probability distribution of the full vocabulary through a small LanguageModelHead (LM-Head): P ti =Softmax(LM-Head(h ti )), P vj =Softmax(LM-Head(h vj )), where P ti ∈R V and P vj ∈R V Denote the vocabulary distribution corresponding to words and frames respectively, and V is the vocabulary size. Then, through the maximum pooling operation, the Top-K points are selected in each distribution: S t =TopK(P ti ), S v =TopK(P vj ), S t and S v They are sparse representations of text and video respectively (each length is equal to the Top-K value). Figure 3 The process of video retrieval enhancement shown in the figure shows that after encoding and inverted index retrieval, the pre-screened videos and their encodings are obtained. These video encodings and text encodings are reordered through the cross-attention module to obtain more accurate retrieval results. In the subsequent processing, the generation module uses these accurately retrieved videos to enhance the generation.

[0049] In addition, the topK value can be dynamically adjusted according to the requirements of the system and task to achieve a balance between performance and efficiency. After constructing the sparse representation, the score of the candidate video can be preliminarily calculated by directly solving the weighted similarity between the sparse representation of the text and the sparse representation of the candidate video, as shown in the following formula (1):

[0050] (1)

[0051] This embodiment of the present invention uses a feature compression method based on a sparsification module to convert the dense representation of the original video into a sparse representation. This method uses sparse vectors to represent video features and constructs an inverted index. Combining the sparse representation with the inverted index enables rapid video retrieval and preliminary screening. Compared to traditional full-data computation methods, the inverted index significantly reduces the amount of retrieval computation, thereby improving recall efficiency and system response speed.

[0052] S4. Through the cross-attention module, based on the query text dense representation and the video dense representation of some candidate videos, determine the global score of each video in some candidate videos, and sort some candidate videos according to the global score.

[0053] Optionally, the structure of the criss-cross attention module is a Transformer layer.

[0054] In one feasible implementation, the module is designed as a small number of Transformer layers that uses cross-attention to capture the semantic alignment between video and text.

[0055] Optionally, the specific operation steps of S4 may include the following steps S41-S42:

[0056] S41. Input the query text dense representation and the video dense representation of some candidate videos into the cross attention module to obtain a global score of the relevance between the input query text and the candidate videos.

[0057] S42. Sort some candidate videos in descending order of global scores.

[0058] In a feasible implementation, the cross attention module is responsible for fine-tuning the sorting of the TOP-N candidate videos after the inverted index is filtered, thereby improving the retrieval accuracy. t ∈R N×d and the dense representation H of the video v ∈R M×d Input to the cross attention module. Its core calculation is as follows: Assume that the i-th word vector after text encoding is h ti , the jth vector of the video frame is h vj , the cross attention is aligned by calculating the attention matrix, as shown in the following formula (2):

[0059] (2)

[0060] Among them, q i =W q h ti ,k j =W k h vj , W q , W k is the query and key linear transformation matrix. Finally, the module outputs a global score of relevance to the input, as shown in Equation (3):

[0061] (3)

[0062] W linear is the linear transformation matrix of the output features. Through the fine reordering of the cross-attention module, the temporal relationship of the video frames and the contextual semantic matching with the text can be more accurately captured.

[0063] This embodiment of the present invention significantly reduces computational complexity while maintaining high retrieval accuracy by combining sparse representation with efficient cross-modal reranking. After quickly screening the top-N candidate videos in the inverted index phase, a cross-attention-based reranking module is introduced to fine-tune the coarse-screened results. This fully captures the contextual semantic connections between video frame information and text descriptions, significantly improving ranking accuracy.

[0064] S5. Generate a target text corresponding to the query text based on the sorted candidate videos and the query text through a generation module.

[0065] Optionally, the generation module adopts a Transformer-based autoregressive generator architecture.

[0066] The specific operation steps of S5 may include the following steps:

[0067] The video information of the sorted candidate videos is deeply integrated with the query text, and the target text corresponding to the query text is gradually generated through a multi-layer decoder based on the Transformer autoregressive generator architecture.

[0068] In one feasible implementation, the generation module aims to deeply fuse the candidate video information obtained through retrieval and re-ranking with the original query text, thereby providing enhanced generation capabilities for tasks such as video question answering, summarization, and multimodal dialogue. This module uses a Transformer-based autoregressive generator architecture to gradually generate the target text through a multi-layer decoder.

[0069] In an embodiment of the present invention, a query text, a candidate video sequence and a trained video retrieval generation model are obtained, and the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module and a generation module; according to the encoder module, the query text and the candidate video sequence are respectively encoded into high-dimensional vector representations to obtain a query text dense representation and a candidate video sequence dense representation; through the sparse representation module, a text sparse representation is generated according to the query text dense representation, and a candidate video sparse representation is generated according to the candidate video sequence dense representation; according to the text sparse representation and the candidate video sparse representation, the candidate video sequence is subjected to inverted index screening to obtain some candidate videos; through the cross-attention module, according to the query text dense representation and the video dense representation of some candidate videos, the global score of each video in some candidate videos is determined, and some candidate videos are sorted according to the global score; through the generation module, according to the sorted candidate videos and the query text, a target text corresponding to the query text is generated. The embodiment of the present invention uses sparse representation and reordering to enable the entire retrieval framework to balance efficiency and performance. It utilizes the RAG system of external video information and uses sparse modules and cross-attention modules to perform rough retrieval and refined reordering, providing the RAG system with low-latency, high-precision video retrieval results, allowing the RAG system to effectively utilize external video information to help generate, achieve a balance between efficiency and performance, and improve the efficiency of video retrieval.

[0070] Figure 4 This is a block diagram of a video retrieval generation device based on sparse representation and reordering provided by an embodiment of the present invention, which is used in a video retrieval generation method based on sparse representation and reordering. Figure 4 The apparatus includes an acquisition unit 410, an encoding unit 420, an inverted index unit 430, a sorting unit 440, and a generation unit 450.

[0071] An acquisition unit 410 is configured to acquire a query text, a candidate video sequence, and a trained video retrieval generation model, wherein the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module, and a generation module;

[0072] An encoding unit 420 is configured to encode the query text and the candidate video sequence into high-dimensional vector representations according to the encoder module to obtain a dense representation of the query text and a dense representation of the candidate video sequence;

[0073] An inverted index unit 430 is configured to generate a text sparse representation based on the query text dense representation, generate a candidate video sparse representation based on the candidate video sequence dense representation, and perform inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos.

[0074] a ranking unit 440 configured to determine a global score of each video in the portion of candidate videos based on the query text dense representation and the video dense representation of the portion of candidate videos using a cross-attention module, and to rank the portion of candidate videos according to the global score;

[0075] The generating unit 450 is configured to generate a target text corresponding to the query text according to the sorted candidate videos and the query text through a generating module.

[0076] Optionally, the encoder module includes a text encoder and a video encoder, the text encoder adopts a Transformer Encoder architecture, and the video encoder adopts a Vision Transformer architecture;

[0077] The encoding unit 420 is configured to:

[0078] S21. Perform word segmentation and embedding on the query text to obtain word vectors, and input the word vectors into the Transformer Encoder layer to obtain a dense representation of the query text.

[0079] S22. Extract multiple key frames from the candidate video sequence by sampling to obtain an input frame set, segment each frame image in the input frame set, obtain an embedded representation based on the segmented image blocks, and input the embedded representation into a video encoder to obtain a dense representation of the candidate video sequence.

[0080] Optionally, the inverted index unit 430 is configured to:

[0081] S31. Projecting the dense representation of the query text onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the word, and selecting a preset number of points in the vocabulary distribution corresponding to the word through the maximum pooling operation to obtain the sparse representation of the text;

[0082] S32. Projecting the dense representation of the candidate video sequence onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the video frame, and selecting a preset number of points in the vocabulary distribution corresponding to the video frame through a maximum pooling operation to obtain a sparse representation of the candidate video;

[0083] S33. Calculate preliminary scores of the candidate videos based on the formaldehyde similarity between the sparse representation of the text and the sparse representation of the candidate videos, sort the multiple preliminary scores from large to small, select the candidate videos corresponding to the preliminary scores in the top N orders, and determine them as some candidate videos.

[0084] The embodiment of the present invention uses sparse representation and reordering to enable the entire retrieval framework to balance efficiency and performance. It utilizes the RAG system of external video information and uses sparse modules and cross-attention modules to perform rough retrieval and refined reordering, providing the RAG system with low-latency, high-precision video retrieval results, so that the RAG system can effectively utilize external video information to help generate and achieve a balance between efficiency and performance.

[0085] Figure 5 is a structural diagram of a video retrieval generation device based on sparse representation and reordering provided by an embodiment of the present invention, such as Figure 5 As shown, the video retrieval generation device based on sparse representation and reordering may include the above Figure 4 Optionally, the video retrieval generation device 510 based on sparse representation and re-ranking may include a first processor 2001 .

[0086] Optionally, the video retrieval generation device 510 based on sparse representation and re-ranking may further include a memory 2002 and a transceiver 2003 .

[0087] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0088] The following combination Figure 5 The components of the video retrieval generation device 510 based on sparse representation and re-ranking are introduced in detail:

[0089] The first processor 2001 is the control center of the sparse representation and reordering-based video retrieval generation device 510 and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), or application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0090] Optionally, the first processor 2001 may execute various functions of the video retrieval generation device 510 based on sparse representation and reordering by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002 .

[0091] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 5 CPU0 and CPU1 are shown in FIG.

[0092] In a specific implementation, as an embodiment, the video retrieval generation device 510 based on sparse representation and re-ranking may also include multiple processors, such as Figure 5 1 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0093] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0094] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, a random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and capable of being accessed by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the video retrieval generation device 510 based on sparse representation and reordering through an interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0095] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0096] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 5The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0097] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be connected to the video retrieval generation device 510 based on sparse representation and reordering via an interface circuit ( Figure 5 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0098] It should be noted that Figure 5 The structure of the video retrieval generation device 510 based on sparse representation and reordering shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0099] In addition, the technical effects of the video retrieval generation device 510 based on sparse representation and reordering can refer to the technical effects of the video retrieval generation method based on sparse representation and reordering described in the above method embodiment, and will not be repeated here.

[0100] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc.

[0101] It should also be understood that the memory in the embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0102] The above embodiments can be implemented in whole or in part via software, hardware (e.g., circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions or computer programs. When loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are fully or partially performed. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired means (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0103] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0104] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0105] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0107] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0108] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0110] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical disks.

[0112] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A video retrieval generation method based on sparse representation and re-ranking, characterized in that: The method comprises: S1: Obtain query text, candidate video sequences, and a trained video retrieval generation model, wherein the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module, and a generation module; S2: According to the encoder module, the query text and candidate video sequence are respectively encoded into high-dimensional vector representations to obtain dense representations of the query text and candidate video sequence; S3: The sparse representation module generates a sparse text representation based on the query text dense representation, and generates a sparse candidate video representation based on the candidate video sequence dense representation. Based on the text sparse representation and the candidate video sparse representation, the candidate video sequence is screened by inverted index to obtain some candidate videos. S4: Determine the global score of each video in the candidate videos based on the query text dense representation and the video dense representation of the candidate videos through the cross attention module, and rank the candidate videos according to the global score; S5: Generate a target text corresponding to the query text based on the sorted candidate videos and the query text through a generation module; The structure of the cross attention module is a Transformer layer; The S4 includes: S41, inputting the query text dense representation and the video dense representation of some candidate videos into a cross attention module to obtain a global score of the relevance between the input query text and the candidate videos; S42. Sort some candidate videos in descending order of global scores; The generation module adopts a Transformer-based autoregressive generator architecture; The S5 includes: The video information of the sorted candidate videos is deeply integrated with the query text, and the target text corresponding to the query text is gradually generated through a multi-layer decoder based on the Transformer autoregressive generator architecture.

2. The video retrieval generation method based on sparse representation and re-ranking according to claim 1 is characterized in that: The encoder module includes a text encoder and a video encoder, the text encoder adopts the Transformer Encoder architecture, and the video encoder adopts the Vision Transformer architecture; The S2, according to the encoder module, encodes the query text and the candidate video sequence into high-dimensional vector representations, respectively, to obtain a dense representation of the query text and a dense representation of the candidate video sequence, including: S21. Perform word segmentation and embedding on the query text to obtain word vectors, and input the word vectors into the TransformerEncoder layer to obtain a dense representation of the query text; S22. Extract multiple key frames from the candidate video sequence by sampling to obtain an input frame set, segment each frame image in the input frame set, obtain an embedded representation based on the segmented image blocks, and input the embedded representation into a video encoder to obtain a dense representation of the candidate video sequence.

3. The video retrieval generation method based on sparse representation and re-ranking according to claim 1, characterized in that: The S3 generates a text sparse representation based on the query text dense representation through a sparse representation module, generates a candidate video sparse representation based on the candidate video sequence dense representation, and performs inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos, including: S31. Projecting the dense representation of the query text onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the word, and selecting a preset number of points in the vocabulary distribution corresponding to the word through the maximum pooling operation to obtain the sparse representation of the text; S32. Projecting the dense representation of the candidate video sequence onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the video frame, and selecting a preset number of points from the vocabulary distribution corresponding to the video frame through a maximum pooling operation to obtain a sparse representation of the candidate video; S33. Calculate preliminary scores of the candidate videos based on the weighted similarity between the sparse representation of the text and the sparse representation of the candidate videos, sort the multiple preliminary scores from large to small, select the candidate videos corresponding to the preliminary scores in the top N orders, and determine them as some candidate videos.

4. A video retrieval generation device based on sparse representation and reordering, wherein the video retrieval generation device based on sparse representation and reordering is used to implement the video retrieval generation method based on sparse representation and reordering according to any one of claims 1 to 3, characterized in that: The device comprises: An acquisition unit, configured to acquire a query text, a candidate video sequence, and a trained video retrieval generation model, wherein the video retrieval generation model includes an encoder module, a sparse representation module, a cross-attention module, and a generation module; An encoding unit, configured to encode the query text and the candidate video sequence into high-dimensional vector representations according to the encoder module, to obtain a dense representation of the query text and a dense representation of the candidate video sequence; An inverted index unit is used to generate a text sparse representation based on the query text dense representation through a sparse representation module, generate a candidate video sparse representation based on the candidate video sequence dense representation, and perform inverted index screening on the candidate video sequence based on the text sparse representation and the candidate video sparse representation to obtain some candidate videos; a ranking unit, configured to determine a global score of each video in the portion of candidate videos based on a dense representation of the query text and dense representations of the portion of candidate videos through a cross-attention module, and to rank the portion of candidate videos according to the global score; The generating unit is configured to generate a target text corresponding to the query text according to the sorted candidate videos and the query text through a generating module.

5. The video retrieval generation device based on sparse representation and reordering according to claim 4, characterized in that: The encoder module includes a text encoder and a video encoder, the text encoder adopts the Transformer Encoder architecture, and the video encoder adopts the Vision Transformer architecture; The encoding unit is used to: S21. Perform word segmentation and embedding on the query text to obtain word vectors, and input the word vectors into the TransformerEncoder layer to obtain a dense representation of the query text; S22. Extract multiple key frames from the candidate video sequence by sampling to obtain an input frame set, segment each frame image in the input frame set, obtain an embedded representation based on the segmented image blocks, and input the embedded representation into a video encoder to obtain a dense representation of the candidate video sequence.

6. The video retrieval generation device based on sparse representation and re-ranking according to claim 4, characterized in that: The inverted index unit is used to: S31. Projecting the dense representation of the query text onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the word, and selecting a preset number of points in the vocabulary distribution corresponding to the word through the maximum pooling operation to obtain the sparse representation of the text; S32. Projecting the dense representation of the candidate video sequence onto the probability distribution of the full vocabulary through the sparse representation module to obtain the vocabulary distribution corresponding to the video frame, and selecting a preset number of points in the vocabulary distribution corresponding to the video frame through a maximum pooling operation to obtain a sparse representation of the candidate video; S33. Calculate preliminary scores of the candidate videos based on the weighted similarity between the sparse representation of the text and the sparse representation of the candidate videos, sort the multiple preliminary scores from large to small, select the candidate videos corresponding to the preliminary scores in the top N orders, and determine them as some candidate videos.

7. A video retrieval generation device based on sparse representation and re-ranking, characterized in that: The video retrieval generation device based on sparse representation and reordering includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 3 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, which can be called by a processor to execute the method according to any one of claims 1 to 3.

Images