Video retrieval method and system based on large language model semantic driving

Abstract

The invention relates to the technical field of text video retrieval, in particular to a video retrieval method and system based on large language model semantic driving, and the method comprises the steps: obtaining a query text and a candidate video set; constructing a text video retrieval model; inputting the candidate video set into a text video retrieval model, and obtaining video features of each candidate video through a video feature extractor; after the query text is embedded into a preset prompt statement, inputting the query text and the video features of all the candidate videos into a large language model, and outputting the similarity between the query text and each candidate video; and outputting a video retrieval result according to the similarity between the query text and each candidate video. According to the method, the video features containing dynamic changes and spatial details are constructed, the cross-modal semantic relationship is obtained by using the powerful semantic reasoning ability of the large language model, the cognitive behavior of human retrieval is met, the interpretability of the model is enhanced, and the precision and efficiency of cross-modal text video retrieval are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text video retrieval, and in particular to a video retrieval method and system based on semantic driving of a large language model. Background Art

[0002] With the popularization of the Internet and the development of digital media technology, video has become an important carrier of information dissemination. The amount of video content has exploded, and how to meet the needs of efficient information retrieval has become an urgent problem to be solved. Text to Video Retrieval is a cross-modal matching task that combines natural language processing and computer vision. Its purpose is to quickly locate and retrieve relevant video clips based on text queries in large-scale video datasets.

[0003] Benefiting from the excellent performance of the Contrastive Language Image Pre-training (CLIP) pre-trained image-text model in image retrieval tasks, some technologies have migrated CLIP to video retrieval tasks and achieved good retrieval results. As a representative technology, the CLIP4Clip model migrates the image encoder and text encoder of CLIP to the video retrieval task, realizing end-to-end video retrieval. First, the technology uniformly samples the video frames and inputs them into the image encoder to extract frame-level features, and uses the text encoder to encode the text description to obtain text features; then, a network structure containing several layers of Transformer modules is used to perform temporal modeling on the frame-level features and average pooling to obtain video features; finally, the cosine similarity of video features and text features is calculated for cross-modal matching, and the model is trained using contrastive learning loss. Many subsequent excellent technologies, such as Cap4Video, HBI, and DRL, have also been improved on this basis, such as auxiliary information enhancement, auxiliary signal supervision, and fine-grained interaction.

[0004] However, the cross-modal matching process in existing text video retrieval technology is in the form of "tokens to tokens", that is, the text is segmented into text tokens such as words, subwords or characters, and the video is segmented into video frames or video blocks as video words. Then, the text tokens and video tokens are extracted separately, and they are converted into fixed-dimensional feature vectors for association and alignment, such as alignment of video frame-level features with text word-level features, alignment of video patch-level features with text word-level features, and alignment of video-level features with text global features. Regardless of which alignment method is used, the semantic correlation between modalities is measured, while the cross-modal semantic reasoning process is ignored, making text video retrieval lack of interpretability and difficult to complete some complex retrieval tasks that require a deep understanding of multimodal semantic information, thereby limiting the accuracy of text video retrieval. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the existing technology, which measures the semantic relevance between modalities but ignores the cross-modal semantic reasoning process, thereby reducing the accuracy of text video retrieval.

[0006] In order to solve the above technical problems, the present invention provides a video retrieval method based on semantic driving of a large language model, comprising: Constructing a text video retrieval model; the text video retrieval model includes a video feature extractor and a large language model; the video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a two-stream cross attention module and a multi-layer perceptron; The candidate video set is input into the text video retrieval model, and the video features of each candidate video are obtained through the video feature extractor, including: The candidate video is passed through a video encoder to output a first frame-level feature and a first patch-level feature; Input the first frame-level feature into the temporal modeling module and output the second frame-level feature; input the first patch-level feature into the spatial modeling module and output the second patch-level feature; Clustering the second patch-level features using a patch-level clustering module to obtain a third patch-level feature; Input the second frame-level feature and the third patch-level feature into the two-stream cross-attention module for interaction, and output the target frame-level feature and the target patch-level feature; Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video; After embedding the query text into the preset prompt sentence, it is input into the large language model together with the video features of all candidate videos, and the similarity between the query text and each candidate video is output; Output video retrieval results based on the similarity between the query text and each candidate video.

[0007] Preferably, the video encoder is constructed based on an image encoder of the CLIP model.

[0008] Preferably, the temporal modeling module and the spatial modeling module are both composed of a stack of several Transformer network layers.

[0009] Preferably, the step of inputting the second frame-level feature and the third patch-level feature into a dual-stream cross attention module for interaction and outputting a target frame-level feature and a target patch-level feature comprises: The dual-stream cross-attention module includes a patch-level cross-attention mechanism and a frame-level cross-attention mechanism; The patch-level cross-attention mechanism takes the third patch-level feature as input query, the second frame-level feature as input key and value, and outputs the target patch-level feature; The frame-level criss-cross attention mechanism takes the second frame-level features as input query, the third patch-level features as input keys and values, and outputs the target frame-level features.

[0010] Preferably, the large language model is an LLaMA model.

[0011] Preferably, when training the text video retrieval model, the parameters of the video encoder and the large language model of the video feature extractor are frozen.

[0012] Preferably, when training the text video retrieval model, a masking and reconstruction module is also included; The mask reconstruction module includes an importance prediction layer, a conditional mask layer and a feature reconstruction layer; The importance prediction layer is used to generate the importance score of each patch-level feature in the second patch-level feature, and includes a first linear layer, a RELU activation layer, a second linear layer and a Softmax layer; wherein the first linear layer is used to map the second patch-level feature from d dimensions to 2*d dimensions for projection transformation; the RELU activation layer is used to maintain the non-negativity of the feature and add nonlinear factors; the second linear layer is used to map the activated feature from 2*d dimensions to 1 dimension; the Softmax layer is used to map the 1-dimensional feature to the (0,1) interval to generate the importance score of each patch-level feature; The formula of the importance prediction layer is expressed as: ; in, and denote the projection parameters of the first and second linear layers respectively, for Activation function, is the Softmax function; represents the second patch-level feature, represents the importance score of the second patch-level feature; The conditional masking layer is used to mask the K features with the highest importance scores of each frame image in the second patch-level features using random noise to obtain the masked patch-level features; The feature reconstruction layer is composed of a stack of encoding layers of several Transformer network layers; in the multi-head attention layer of the encoding layer of each Transformer network layer, the masked patch-level features are used as input queries, the second frame-level features are used as input keys and values, the masked patch-level features are reconstructed, and the reconstructed patch-level features are output, which is expressed as follows: ; in, represents the feature reconstruction module, , and are the linear transformation parameters for query, key, and value, respectively; represents the reconstructed patch-level features, represents the patch-level features after masking, Represents the second frame level feature.

[0013] Preferably, the loss function of the text video retrieval model includes a contrastive learning loss function and a reconstruction loss function; the reconstruction loss function is used to calculate the difference between the reconstructed patch-level feature and the second patch-level feature.

[0014] Preferably, the patch-level clustering module clusters the second patch-level features based on a Kmeans clustering algorithm.

[0015] The present invention also provides a video retrieval system based on a large language model semantic drive, comprising: A retrieval model building module, used to build a text video retrieval model; the text video retrieval model includes a video feature extractor and a large language model; the video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a dual-stream cross attention module and a multi-layer perceptron; The feature extraction module is used to input the candidate video set into the text video retrieval model and obtain the video features of each candidate video through the video feature extractor, including: The candidate video is passed through a video encoder to output a first frame-level feature and a first patch-level feature; Input the first frame-level feature into the temporal modeling module and output the second frame-level feature; input the first patch-level feature into the spatial modeling module and output the second patch-level feature; Clustering the second patch-level features using a patch-level clustering module to obtain a third patch-level feature; Input the second frame-level feature and the third patch-level feature into the two-stream cross-attention module for interaction, and output the target frame-level feature and the target patch-level feature; Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video; A similarity acquisition module is used to embed the query text into a preset prompt sentence, input the query text and the video features of all candidate videos into the large language model, and output the similarity between the query text and each candidate video; The retrieval output module is used to output the video retrieval results based on the similarity between the query text and each candidate video.

[0016] The above technical solution of the present invention has the following beneficial effects compared with the prior art: The video retrieval method based on the semantic drive of a large language model described in the present invention extracts the frame-level features and patch-level features of the candidate video, and performs temporal modeling on the frame-level features to capture the changes of the video content on the time axis, which is helpful to extract the temporal dynamics of the video, understand the dynamic content of the video, perform spatial modeling and clustering on the patch-level features, capture the spatial relationship between the parts of the video screen, and help extract the spatial structure distribution of the video screen; further based on the cross-attention mechanism, the frame-level features and the patch-level features are multi-granularly interacted, which can integrate the information of the video in the two dimensions of time and space, construct video features containing dynamic changes and spatial details, and retain the key information of the video to the greatest extent, and provide rich and accurate video information for the text video retrieval task, so as to improve the accuracy and efficiency of the text video retrieval. In addition, the present invention inputs the extracted video features and the prompt sentence embedded in the query text into the large language model for semantic reasoning and cross-modal semantic matching, and uses the powerful semantic reasoning ability of the large language model to obtain a more reliable and more explanatory cross-modal semantic relationship, which conforms to the cognitive behavior of human retrieval, enhances the interpretability of the model, and improves the accuracy and efficiency of cross-modal text video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1 It is a flow chart of a video retrieval method based on semantic driving of a large language model according to the present invention. DETAILED DESCRIPTION

[0018] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention. Embodiment 1

[0019] Reference Figure 1 As shown, the present invention provides a video retrieval method based on semantic driving of a large language model, comprising: S1. Obtain query text and candidate video set.

[0020] S2. Build a text-video retrieval model.

[0021] The text video retrieval model includes a video feature extractor and a large language model.

[0022] The video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a two-stream cross-attention module and a multi-layer perceptron.

[0023] S3. Input the candidate video set into the text video retrieval model, and obtain the video features of each candidate video through the video feature extractor, including: S3-1. The candidate video outputs the first frame-level features and the first patch-level features through the video encoder.

[0024] Preferably, this embodiment constructs a video encoder based on the image encoder of the CLIP model, and the video encoder is composed of 12 stacked Transformer network layers.

[0025] Specifically, the input of the video encoder is a video frame image uniformly extracted from the candidate video. ,in Indicates the number of sampled frames. Each frame is divided into patches of equal size and non-overlapping. After simple encoding, a classification tag [CLS] token is added to each frame and then input into the video encoder.

[0026] The video encoder performs attention operation on each frame image and outputs the first frame level features ; Perform attention operation on the patch after segmentation of each frame image and output the first patch level feature ,in, represents the m patch-level features of the first frame image, represents the m patch-level features of the nth frame image; m represents the number of non-overlapping patches into which each frame image is divided, Represents the dimension of the feature.

[0027] S3-2, input the first frame-level feature into the temporal modeling module, and output the second frame-level feature; input the first patch-level feature into the spatial modeling module, and output the second patch-level feature.

[0028] Preferably, the time series modeling module is composed of a plurality of stacked Transformer network layers. In this embodiment, the number of the Transformer network layers in the time series modeling module is 4.

[0029] The temporal modeling module is based on the first frame level features Output the second frame-level features after corresponding temporal modeling .

[0030] Preferably, the spatial modeling module is composed of a plurality of stacked Transformer network layers. In this embodiment, the number of the Transformer network layers in the spatial modeling module is 4.

[0031] The spatial modeling module is based on the first patch level features Output the corresponding second patch-level features after spatial modeling .

[0032] S3-3. Cluster the second patch-level features using a patch-level clustering module to obtain third patch-level features.

[0033] Preferably, the patch-level clustering module clusters the second patch-level features based on a Kmeans clustering algorithm.

[0034] The patch-level clustering module uses the second patch-level features Clustering is performed, and the m patches in each frame image are clustered to generate o cluster centers, which are then spliced ​​to obtain the third patch-level feature ,in, represents the patch-level features of o clusters in the first frame image, Represents the patch-level features of o clusters of the n-th frame image; o represents the number of cluster centers.

[0035] S3-4. Input the second frame-level features and the third patch-level features into the dual-stream cross-attention module for multi-granularity interaction, and output the target frame-level features and the target patch-level features.

[0036] Specifically, the two-stream cross-attention module includes a patch-level cross-attention mechanism and a frame-level cross-attention mechanism, which is used to and third patch level features Interact to generate target frame-level features and target patch-level features .

[0037] Both the patch-level cross-attention mechanism and the frame-level cross-attention mechanism include a feed-forward linear layer and a multi-head attention. The feed-forward linear layer is used to adjust the dimension and distribution of the input data, convert the input data into a vector representation, and provide a suitable input for the subsequent multi-head attention mechanism. The multi-head attention is used to enhance the feature representation of the input data.

[0038] Specifically, in the patch-level cross attention mechanism, the third patch-level feature As input query , the second frame level feature As input key Sum , output target patch-level features .

[0039] In the frame-level cross attention mechanism, the second frame-level feature As input query , the third patch level feature As input key Sum , output target frame-level features .

[0040] S3-5. Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video.

[0041] Specifically, this embodiment builds a multi-layer perceptron based on a fully connected network. The multi-layer perceptron fuses the target frame-level features. and target patch-level features Output video characteristics .

[0042] S4: After embedding the query text into the preset prompt sentence, it is input into the large language model together with the video features of all candidate videos, and the similarity between the query text and each candidate video is output.

[0043] The prompt statement is used to obtain the input of the large language model according to the query text, and guide the large language model to calculate the similarity of each video in the candidate video set in descending order according to the relevance between the candidate video and the query text, and limit the range of the similarity to between 0 and 1, and output it in the form of a list.

[0044] In this embodiment, the preset prompt sentence is set as: Please compute the similarities (ranging from 0 to 1) of the videos in the video collection in descending order according to their relevance to the sentence " " and output them as alist; After obtaining the query text, it is embedded into the " " position of the preset prompt sentence to obtain the input of the large language model.

[0045] Preferably, the large language model used in this embodiment is an LLaMA model.

[0046] Specifically, the large language model is based on the prompt sentence embedded in the query text and the video features of all candidate videos. Perform semantic-driven reasoning and output the similarity between the query text and each candidate video.

[0047] S5. Output the video retrieval results based on the similarity between the query text and each candidate video.

[0048] Preferably, when training the text video retrieval model, a masking and reconstruction module is also included.

[0049] The masking and reconstruction module is used in the model training stage, and includes an importance prediction layer, a conditional masking layer and a feature reconstruction layer.

[0050] Specifically, the importance prediction layer includes a first linear layer, a RELU activation layer, a second linear layer and a Softmax layer for predicting the importance of the second patch level feature. , generates a non-negative importance score for each patch-level feature in the second patch-level feature Among them, the first linear layer is used to map the second patch-level features from d dimensions to 2*d dimensions for projection transformation; the RELU activation layer is used to maintain the non-negativity of the features and add nonlinear factors to increase the nonlinear expression ability of the importance prediction layer; the second linear layer is used to map the activated features from 2*d dimensions to 1 dimension; the Softmax layer is used to map the 1-dimensional features to The interval indicates the importance of the feature.

[0051] The formula of the importance prediction layer is expressed as: ; in, and denote the projection parameters of the first and second linear layers respectively, for Activation function, is the Softmax function; represents the second patch-level feature, Represents the importance score of the second patch-level feature.

[0052] The conditional masking layer is based on the non-negative importance score The second patch-level feature is conditionally masked, specifically: the K features with the highest importance scores of each frame image in the second patch-level feature are masked with random noise to obtain the masked patch-level feature ,in, represents the masked feature of the jth patch in the first frame image, Represents the masked features of the kth patch in the i-th frame image.

[0053] The feature reconstruction layer is based on the second frame level features Reconstruct the masked patch-level features.

[0054] Specifically, the feature reconstruction layer is composed of a stack of encoding layers of several Transformer network layers, and the core of the encoding layer of each Transformer network layer is a multi-head attention layer. In the multi-head attention layer, the masked patch-level features are used as input queries. , using the second frame-level features as input keys Sum The output of the feature reconstruction module is the reconstructed patch-level features .

[0055] The formula of the feature reconstruction layer is expressed as: ; in, represents the feature reconstruction module, , and are the linear transformation parameters for query, key, and value, respectively; represents the reconstructed patch-level features, represents the patch-level features after masking, Represents the second frame level feature.

[0056] The masking reconstruction module reconstructs the masked patch-level features using the second frame-level features to enhance the representation capability of the second frame-level features during training.

[0057] In this embodiment, the process of training the text video retrieval model includes: (1) Obtain the video-text pair data in the training set, initialize the model parameters, and set the learning rate, decay strategy, and number of iterations.

[0058] Specifically, this embodiment uses official data division to obtain a training set; use Initialize the video encoder, Initialize the large language model, The method initializes the parameters of the remaining modules based on a normal distribution; the learning rate is set to ; Select the cosine decay strategy; Set the number of iterations to 5.

[0059] (2) n video-text pairs are batch-inputted into the pre-built text-video retrieval model. Fixed-length frame-level features and patch-level features are extracted from each video. The video features extracted by the video feature extractor and the prompt sentence embedded in the query text are input into the large language model together. The output size is The similarity matrix of .

[0060] (3) The total loss function is constructed using contrastive learning loss and reconstruction loss function to supervise the training of the text video retrieval model. The model is iteratively optimized through back propagation according to the pre-set learning rate to obtain the trained text video retrieval model.

[0061] Specifically, the contrastive learning loss function obtains the loss value according to the similarity matrix output by the large language model. Through back propagation, the model parameters are updated to increase the similarity between positive samples and reduce the similarity between negative samples. Its formula is expressed as: ; in, represents the contrastive learning loss function, represents the batch size, Represents the similarity of the output of the large language model, and Respectively represent the first The text and Videos, and Respectively represent the first The text and Videos; Represents the temperature coefficient.

[0062] The reconstruction loss function is calculated by calculating the reconstructed patch-level features and second patch level features The difference between them generates a loss value, and the model parameters are updated through back propagation to reduce the difference between the two sets of patch-level features and maintain the consistency between the reconstructed patch-level features and the second patch-level features. The formula is expressed as: ; in, represents the reconstruction loss function.

[0063] Therefore, the total loss function of the training process is It is expressed as: ,in represents the equilibrium parameter.

[0064] Preferably, when training the text video retrieval model, the parameters of the video encoder and the large language model of the video feature extractor are frozen, and only the parameters of the video feature extractor are iteratively updated, thereby reducing the training cost to a certain extent.

[0065] In summary, the video retrieval method based on the semantic drive of a large language model described in the present invention extracts the frame-level features and patch-level features of the candidate video, and performs temporal modeling on the frame-level features, captures the changes of the video content on the time axis, helps to extract the temporal dynamics of the video, understands the dynamic content of the video, performs spatial modeling and clustering on the patch-level features, captures the spatial relationship between the parts of the video screen, and helps to extract the spatial structure distribution of the video screen; further based on the cross-attention mechanism, the frame-level features and the patch-level features are multi-granularly interacted, and the information of the video in the two dimensions of time and space can be integrated, and the video features containing dynamic changes and spatial details can be constructed, and the key information of the video can be retained to the greatest extent, and rich and accurate video information is provided for the text video retrieval task to improve the accuracy and efficiency of the text video retrieval. In addition, the present invention inputs the extracted video features and the prompt sentence embedded in the query text into the large language model for semantic reasoning and cross-modal semantic matching, and uses the powerful semantic reasoning ability of the large language model to obtain a more reliable and more explanatory cross-modal semantic relationship, which conforms to the cognitive behavior of human retrieval, enhances the interpretability of the model, and improves the accuracy and efficiency of cross-modal text video retrieval. Embodiment 2

[0066] Based on the video retrieval method based on large language model semantics drive described in the first embodiment, this embodiment provides a video retrieval system based on large language model semantics drive, including: A retrieval model building module, used to build a text video retrieval model; the text video retrieval model includes a video feature extractor and a large language model; the video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a dual-stream cross attention module and a multi-layer perceptron; The feature extraction module is used to input the candidate video set into the text video retrieval model and obtain the video features of each candidate video through the video feature extractor, including: The candidate video is passed through a video encoder to output a first frame-level feature and a first patch-level feature; Input the first frame-level feature into the temporal modeling module and output the second frame-level feature; input the first patch-level feature into the spatial modeling module and output the second patch-level feature; Clustering the second patch-level features using a patch-level clustering module to obtain a third patch-level feature; Input the second frame-level feature and the third patch-level feature into the two-stream cross-attention module for interaction, and output the target frame-level feature and the target patch-level feature; Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video; A similarity acquisition module is used to embed the query text into a preset prompt sentence, input the query text and the video features of all candidate videos into the large language model, and output the similarity between the query text and each candidate video; The retrieval output module is used to output the video retrieval results based on the similarity between the query text and each candidate video.

[0067] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0068] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0071] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A video retrieval method based on semantics driven by a large language model, characterized in that: include: Constructing a text video retrieval model; the text video retrieval model includes a video feature extractor and a large language model; the video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a two-stream cross attention module and a multi-layer perceptron; The candidate video set is input into the text video retrieval model, and the video features of each candidate video are obtained through the video feature extractor, including: The candidate video is passed through a video encoder to output a first frame-level feature and a first patch-level feature; Input the first frame-level feature into the temporal modeling module and output the second frame-level feature; input the first patch-level feature into the spatial modeling module and output the second patch-level feature; Clustering the second patch-level features using a patch-level clustering module to obtain a third patch-level feature; Input the second frame-level feature and the third patch-level feature into the two-stream cross-attention module for interaction, and output the target frame-level feature and the target patch-level feature; Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video; After embedding the query text into the preset prompt sentence, it is input into the large language model together with the video features of all candidate videos, and the similarity between the query text and each candidate video is output; Output video retrieval results based on the similarity between the query text and each candidate video.

2. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: The video encoder is built based on the image encoder of the CLIP model.

3. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: The temporal modeling module and the spatial modeling module are both composed of a stack of several Transformer network layers.

4. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: The second frame-level feature and the third patch-level feature are input into the dual-stream cross attention module for interaction, and the target frame-level feature and the target patch-level feature are output, including: The dual-stream cross-attention module includes a patch-level cross-attention mechanism and a frame-level cross-attention mechanism; The patch-level cross-attention mechanism takes the third patch-level feature as input query, the second frame-level feature as input key and value, and outputs the target patch-level feature; The frame-level criss-cross attention mechanism takes the second frame-level features as input query, the third patch-level features as input keys and values, and outputs the target frame-level features.

5. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: The large language model is an LLaMA model.

6. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: When training the text video retrieval model, the parameters of the video encoder and the large language model of the video feature extractor are frozen.

7. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: When training the text video retrieval model, a masking and reconstruction module is also included; The mask reconstruction module includes an importance prediction layer, a conditional mask layer and a feature reconstruction layer; The importance prediction layer is used to generate the importance score of each patch-level feature in the second patch-level feature, and includes a first linear layer, a RELU activation layer, a second linear layer and a Softmax layer; wherein the first linear layer is used to map the second patch-level feature from d dimensions to 2*d dimensions for projection transformation; the RELU activation layer is used to maintain the non-negativity of the feature and add nonlinear factors; the second linear layer is used to map the activated feature from 2*d dimensions to 1 dimension; the Softmax layer is used to map the 1-dimensional feature to the interval (0,1) to generate the importance score of each patch-level feature; The formula of the importance prediction layer is expressed as: ； in, and denote the projection parameters of the first and second linear layers respectively, for Activation function, is the Softmax function; represents the second patch-level feature, represents the importance score of the second patch-level feature; The conditional masking layer is used to mask the K features with the highest importance scores of each frame image in the second patch-level features using random noise to obtain the masked patch-level features; The feature reconstruction layer is composed of a stack of encoding layers of several Transformer network layers; in the multi-head attention layer of the encoding layer of each Transformer network layer, the masked patch-level features are used as input queries, the second frame-level features are used as input keys and values, the masked patch-level features are reconstructed, and the reconstructed patch-level features are output, which is expressed as follows: ； in, represents the feature reconstruction module, , and are the linear transformation parameters for query, key, and value, respectively; represents the reconstructed patch-level features, represents the patch-level features after masking, Represents the second frame level feature.

8. The video retrieval method based on semantics driven by a large language model according to claim 7, characterized in that: The loss function of the text video retrieval model includes a contrastive learning loss function and a reconstruction loss function; the reconstruction loss function is used to calculate the difference between the reconstructed patch-level feature and the second patch-level feature.

9. The video retrieval method based on semantics driven by a large language model according to claim 1, characterized in that: The patch-level clustering module clusters the second patch-level features based on a Kmeans clustering algorithm.

10. A video retrieval system based on semantics driven by a large language model, characterized in that: include: A retrieval model building module, used to build a text video retrieval model; the text video retrieval model includes a video feature extractor and a large language model; the video feature extractor includes a video encoder, a temporal modeling module, a spatial modeling module, a patch-level clustering module, a dual-stream cross attention module and a multi-layer perceptron; The feature extraction module is used to input the candidate video set into the text video retrieval model and obtain the video features of each candidate video through the video feature extractor, including: The candidate video is passed through a video encoder to output a first frame-level feature and a first patch-level feature; Input the first frame-level feature into the temporal modeling module and output the second frame-level feature; input the first patch-level feature into the spatial modeling module and output the second patch-level feature; Clustering the second patch-level features using a patch-level clustering module to obtain a third patch-level feature; Input the second frame-level feature and the third patch-level feature into the two-stream cross-attention module for interaction, and output the target frame-level feature and the target patch-level feature; Input the target frame-level features and the target patch-level features into a multi-layer perceptron, and output the video features of the candidate video; A similarity acquisition module is used to embed the query text into a preset prompt sentence, input the query text and the video features of all candidate videos into the large language model, and output the similarity between the query text and each candidate video; The retrieval output module is used to output the video retrieval results based on the similarity between the query text and each candidate video.

Images