Video retrieval method and device, electronic equipment and storage medium

Abstract

The invention discloses a video retrieval method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring each current to-be-retrieved video; dividing each video to be retrieved into a plurality of split videos; for each target retrieval dimension, obtaining a text description of the target retrieval dimension of each split video; wherein the target retrieval dimension comprises a subtitle dimension, an audio dimension and a visual dimension; the text description of the visual dimension of one split video is combined by the context text description of the split video and the text description of each key frame of the split video; combining the text description of the target retrieval dimension of each split video with a retrieval problem, and performing retrieval enhancement to obtain a plurality of retrieval videos based on the target retrieval dimension; weighting the text description of each target retrieval dimension of each retrieval video to obtain a plurality of sub-mirror multi-dimensional text descriptions; and combining the multi-dimensional text description of each sub-mirror with a retrieval problem to carry out retrieval enhancement to obtain a final retrieval result.

Description

Technical Field

[0001] The present application relates to the technical field of information retrieval, and particularly relates to a video retrieval method and apparatus, an electronic device, and a storage medium. Background Art

[0002] In recent years, text-video retrieval technology has made remarkable progress. Specifically, it realizes video retrieval by capturing the correlation between text and visual features of videos. Compared with the previous method of retrieving based on text annotation information of videos, it can retrieve the required videos more accurately.

[0003] Currently, for the method of text-video retrieval, the video is mainly divided into single frames or multiple video segments, and then the features of each video frame or each video segment are extracted through a model. Finally, the features of each video frame or each video segment of the video are matched with the retrieval text to implement video retrieval, so as to retrieve a video with a relatively high matching degree with the retrieval problem as a result for output.

[0004] However, the information considered by this method is limited, only considering the features in the visual dimension, which will affect the retrieval accuracy. Although there are currently some methods that consider multiple dimensions for retrieval at the same time, they only simply combine this information and then use the combined feature information for retrieval. Therefore, its retrieval accuracy is also relatively limited. Moreover, since the current method only considers the information of single-frame videos or video segments, it will lead to the loss of the overall context information of the video or fine-grained information, and there are also video frames or video segments with low information in the considered information, thus affecting the retrieval accuracy. Therefore, the accuracy of the current video retrieval method is still relatively low. Summary of the Invention

[0005] Based on the above deficiencies of the prior art, the present application provides a video retrieval method and apparatus, an electronic device, and a storage medium to solve the problem that the prior art cannot effectively guarantee the accuracy of video retrieval.

[0006] To achieve the above object, the present application provides the following technical solutions:

[0007] The first aspect of the present application provides a video retrieval method, including:

[0008] Obtain current videos to be retrieved;

[0009] Divide each of the videos to be retrieved into multiple storyboard videos;

[0010] For each target retrieval dimension, obtain the text descriptions of the target retrieval dimension for each of the sub-shot videos; wherein, the target retrieval dimensions include a subtitle dimension, an audio dimension, and a visual dimension; the text description of the visual dimension of a sub-shot video is composed of the context text description of the sub-shot video and the text descriptions of the key frames of each frame of the sub-shot video that are filtered out;

[0011] Combine the text descriptions of the target retrieval dimension of each of the sub-shot videos with the retrieval question to perform retrieval enhancement for the target retrieval dimension, and obtain multiple retrieval videos based on the target retrieval dimension;

[0012] Weight the text descriptions of each of the target retrieval dimensions of each of the retrieval videos to obtain multiple sub-shot multi-dimensional text descriptions;

[0013] Combine each of the sub-shot multi-dimensional text descriptions with the retrieval question to perform retrieval enhancement, and obtain the final retrieval result.

[0014] Optionally, in the above video retrieval method, when the target retrieval dimension is the subtitle dimension, the obtaining the text descriptions of the target retrieval dimension for each of the sub-shot videos includes:

[0015] Perform character recognition on each video frame of each of the sub-shot videos through character recognition technology to obtain a sub-shot subtitle content library composed of the subtitle contents of each of the sub-shot videos.

[0016] Optionally, in the above video retrieval method, when the target retrieval dimension is the visual dimension, the obtaining the text descriptions of the target retrieval dimension for each of the sub-shot videos includes:

[0017] Use a multi-modal large model to perform description analysis on each of the sub-shot videos to obtain the context text descriptions of each of the sub-shot videos, and use the multi-modal large model to perform description analysis on each video frame of each of the sub-shot videos to obtain the text descriptions of each video frame of each of the sub-shot videos;

[0018] Use a pre-trained information quantity scoring model to score each video frame of each of the sub-shot videos;

[0019] Based on the scores of each video frame, screen out the key frames of each frame of each of the sub-shot videos from each video frame;

[0020] Combine the context text description of each of the sub-shot videos and the text descriptions of the key frames of each frame of the sub-shot video respectively to obtain a sub-shot visual multi-granularity text description library composed of each sub-shot visual multi-granularity text description.

[0021] Optionally, in the above video retrieval method, the step of using the pre-trained information quantity score model to score each video frame of each sub-shot video includes:

[0022] For each video frame of each sub-shot video, input the video frame into the image encoding module in the pre-trained information quantity score model; wherein, the information quantity score model includes an image encoding module, a text encoding module, and a loss function, and they are jointly trained;

[0023] Extract spatio-temporal domain features from the video frame through the feature pre-processor in the image encoding module;

[0024] Use the attention-based discrete wavelet transform method to transform the spatio-temporal domain features into the frequency domain space for attention mechanism learning, and obtain an attention frequency domain space matrix;

[0025] Perform inverse transformation on the attention frequency domain space matrix through the inverse discrete wavelet transform method to obtain an attention-based transformation feature matrix;

[0026] Perform fine-grained separation on the attention-based transformation feature matrix through a fine-grained separation module based on the frequency domain space to obtain a fine-grained frequency domain feature matrix;

[0027] Output the score of the video frame through the information quantity score layer based on the fine-grained frequency domain feature matrix.

[0028] Optionally, in the above video retrieval method, when the target retrieval dimension is the audio dimension, the step of obtaining the text description of the target retrieval dimension of each sub-shot video includes:

[0029] Extract the audio data of each sub-shot video;

[0030] Use the audio language large model to extract information from the audio data of each sub-shot video to obtain a sub-shot audio information text description library composed of text descriptions of each sub-shot audio information.

[0031] Optionally, in the above video retrieval method, the step of combining the text description of the target retrieval dimension of each sub-shot video with the retrieval question to perform retrieval enhancement in the target retrieval dimension to obtain multiple retrieval videos based on the target retrieval dimension includes:

[0032] Convert the text description of the target retrieval dimension of each sub-shot video into a feature vector of the target retrieval dimension of each sub-shot video;

[0033] Based on the feature vectors of the target retrieval dimensions of each of the segmented videos and the feature vector of the retrieval problem, multiple feature vectors of the target retrieval dimensions similar to the retrieval problem are screened out;

[0034] The text descriptions of each of the target retrieval dimensions corresponding to the screened-out feature vectors of each of the target retrieval dimensions are combined with the prompt information and the retrieval problem, and input into a large language model for retrieval enhancement to obtain multiple retrieval videos based on the target retrieval dimensions.

[0035] Optionally, in the above video retrieval method, the combining the text descriptions of each of the segmented multi-dimensional texts with the retrieval problem for retrieval enhancement to obtain the final retrieval result includes:

[0036] Based on the feature vectors of the text descriptions of each of the segmented multi-dimensional texts and the feature vector of the retrieval problem, multiple text descriptions of the segmented multi-dimensional texts similar to the retrieval problem are screened out;

[0037] The screened-out text descriptions of each of the segmented multi-dimensional texts are combined with the prompt information and the retrieval problem, and input into a large language model for retrieval enhancement to obtain the final retrieval result.

[0038] A second aspect of the present application provides a video retrieval device, including:

[0039] A video acquisition unit for acquiring each of the current videos to be retrieved;

[0040] A segmentation unit for respectively dividing each of the videos to be retrieved into multiple segmented videos;

[0041] A text description acquisition unit for respectively obtaining, for each target retrieval dimension, the text description of the target retrieval dimension of each of the segmented videos; wherein, the target retrieval dimensions include a subtitle dimension, an audio dimension, and a visual dimension; the text description of the visual dimension of a segmented video is combined by the context text description of the segmented video and the text descriptions of the key frames of each frame of the screened-out segmented video;

[0042] A preliminary retrieval unit for combining the text descriptions of the target retrieval dimensions of each of the segmented videos with the retrieval problem to perform retrieval enhancement of the target retrieval dimension to obtain multiple retrieval videos based on the target retrieval dimension;

[0043] A fusion unit for weighting the text descriptions of each of the target retrieval dimensions of each of the retrieval videos to obtain multiple segmented multi-dimensional text descriptions;

[0044] A final retrieval unit for combining each of the segmented multi-dimensional text descriptions with the retrieval question for retrieval enhancement to obtain a final retrieval result.

[0045] Optionally, in the above video retrieval device, when the target retrieval dimension is the subtitle dimension, the text description acquisition unit executes the acquisition of the text description of the target retrieval dimension of each of the segmented videos for:

[0046] Performing character recognition on each video frame of each of the segmented videos through character recognition technology to obtain a segmented subtitle content library composed of the subtitle content of each of the segmented videos.

[0047] Optionally, in the above video retrieval device, when the target retrieval dimension is the visual dimension, the text description acquisition unit executes the acquisition of the text description of the target retrieval dimension of each of the segmented videos for:

[0048] Using a multi-modal large model to perform descriptive analysis on each of the segmented videos to obtain the context text description of each of the segmented videos, and using the multi-modal large model to perform descriptive analysis on each video frame of each of the segmented videos to obtain the text description of each video frame of each of the segmented videos;

[0049] Using a pre-trained information quantity scoring model to score each video frame of each of the segmented videos;

[0050] Based on the scores of each video frame, screening out the key frames of each of the segmented videos from each video frame;

[0051] Combining the context text description of each of the segmented videos and the text description of each key frame of the segmented videos respectively to obtain a segmented visual multi-granularity text description library composed of the segmented visual multi-granularity text descriptions of each.

[0052] Optionally, in the above video retrieval device, when the text description acquisition unit executes the scoring of each video frame of each of the segmented videos using the pre-trained information quantity scoring model, it is used for:

[0053] For each video frame of each of the segmented videos respectively, inputting the video frame into the image encoding module in the pre-trained information quantity scoring model; wherein, the information quantity scoring model includes an image encoding module, a text encoding module, and a loss function, and is jointly trained;

[0054] Extracting spatio-temporal domain features from the video frame through the feature pre-processor in the image encoding module;

[0055] Using the attention-based discrete wavelet transform method, convert the spatio-temporal domain features to the frequency domain space for attention mechanism learning to obtain an attention frequency domain space matrix;

[0056] Perform inverse transformation on the attention frequency domain space matrix through the inverse discrete wavelet transform method to obtain an attention-based transformation feature matrix;

[0057] Perform fine-grained separation on the attention-based transformation feature matrix through a fine-grained separation module based on the frequency domain space to obtain a fine-grained frequency domain feature matrix;

[0058] Output the score of the video frame through the information quantity scoring layer based on the fine-grained frequency domain feature matrix.

[0059] Optionally, in the above video retrieval device, when the target retrieval dimension is the audio dimension, the text description acquisition unit executes the acquisition of the text description of the target retrieval dimension of each of the sub-shot videos for:

[0060] Extract the audio data of each of the sub-shot videos;

[0061] Use the audio language large model to extract information from the audio data of each of the sub-shot videos to obtain a sub-shot audio information text description library composed of the text descriptions of each sub-shot audio information.

[0062] Optionally, in the above video retrieval device, the preliminary retrieval unit includes:

[0063] A first conversion unit for converting the text description of the target retrieval dimension of each of the sub-shot videos into a feature vector of the target retrieval dimension of each of the sub-shot videos;

[0064] A first screening unit for screening out multiple feature vectors of the target retrieval dimension similar to the retrieval problem based on the feature vectors of the target retrieval dimension of each of the sub-shot videos and the feature vector of the retrieval problem;

[0065] A first retrieval unit for combining the text descriptions of the target retrieval dimension corresponding to the screened feature vectors of each of the target retrieval dimensions with the prompt information and the retrieval problem, and inputting them into a large language model for retrieval enhancement to obtain multiple retrieval videos based on the target retrieval dimension.

[0066] Optionally, in the above video retrieval device, the final retrieval unit includes:

[0067] A second screening unit for screening out multiple sub-shot multi-dimensional text descriptions similar to the retrieval problem based on the feature vectors of each of the sub-shot multi-dimensional text descriptions and the feature vector of the retrieval problem;

[0068] A second retrieval unit for combining the screened multi-dimensional text descriptions of each sub-shot with the prompt information and the retrieval question, inputting them into a large language model for retrieval enhancement, and obtaining the final retrieval result.

[0069] A third aspect of the present application provides an electronic device, including:

[0070] A memory and a processor;

[0071] Wherein, the memory is used for storing a program;

[0072] The processor is used for executing the program, and when the program is executed, it is specifically used to implement the video retrieval method as described in any one of the above.

[0073] A fourth aspect of the present application provides a computer storage medium for storing a computer program, which is used to implement the video retrieval method as described in any one of the above when executed by a processor.

[0074] The present application provides a video retrieval method. Current videos to be retrieved are obtained, and each video to be retrieved is divided into multiple sub-shot videos respectively for retrieval in terms of the dimensions of sub-shots, ensuring the accuracy of retrieval. For each target retrieval dimension respectively, text descriptions of the target retrieval dimension of each sub-shot video are obtained, and the text descriptions of the target retrieval dimension of each sub-shot video are combined with the retrieval question to perform retrieval enhancement for the target retrieval dimension, obtaining multiple retrieved videos based on the target retrieval dimension. Among them, the target retrieval dimensions include a subtitle dimension, an audio dimension, and a visual dimension, so as to perform precise video retrieval from three dimensions by extracting text descriptions of the subtitle, audio, and visual dimensions. And, the text description of the visual dimension of a sub-shot video is combined by the context text description of the sub-shot video and the text descriptions of the key frames of each frame of the screened sub-shot video, thus fully considering the context information of the video, also considering the fine-grained information of each video frame, and filtering out the information of low-information video frames to avoid affecting the accuracy of the retrieval result, effectively ensuring the accuracy of the retrieval dimension in the visual dimension. Finally, the text descriptions of each target retrieval dimension of each retrieved video are weighted to obtain multiple multi-dimensional text descriptions of sub-shots, and the multi-dimensional text descriptions of each sub-shot are combined with the retrieval question for retrieval enhancement to obtain the final retrieval result, so as to further fuse the features of the subtitle, audio, and visual dimensions on the basis of the retrieved videos retrieved in the three target retrieval dimensions and perform further retrieval, thus effectively ensuring the accuracy of the retrieval. Description of the Drawings

[0075] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on the provided drawings.

[0076] Figure 1 The flowchart of a video retrieval method provided by an embodiment of the present application;

[0077] Figure 2 The flowchart of a method for obtaining text descriptions of visual dimensions of each sub-shot video provided by an embodiment of the present application;

[0078] Figure 3 The flowchart of a method for scoring each video frame of each sub-shot video using an information amount scoring model provided by an embodiment of the present application;

[0079] Figure 4 The flowchart of a method for retrieving based on a target retrieval dimension provided by an embodiment of the present application;

[0080] Figure 5 The schematic architecture diagram of a video retrieval device provided by an embodiment of the present application;

[0081] Figure 6 The schematic architecture diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0082] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0083] In this application, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0084] An embodiment of this application provides a video retrieval method, as Figure 1 shown, which specifically includes the following steps:

[0085] S101. Obtain each current video to be retrieved.

[0086] Optionally, each current video to be retrieved can specifically be obtained by matching the retrieval problem directly input by the user with the attribute information of each video in the video library, so as to efficiently retrieve the coarse-grained videos related to the retrieval problem, in order to reduce the number of videos for subsequent analysis, and thus effectively improve the retrieval efficiency. Of course, it can also be determined by other means for each current video to be retrieved.

[0087] S102. Divide each video to be retrieved into multiple segmented videos respectively.

[0088] In order to perform video retrieval more precisely, and since the content of each segmented video is different, in the embodiment of this application, each video to be retrieved is divided into multiple segments, so as to perform retrieval at the granularity of the segments.

[0089] S103. For each target retrieval dimension, obtain the text description of each segmented video in this target retrieval dimension.

[0090] Among them, the target retrieval dimensions include the subtitle dimension, the audio dimension and the visual dimension. In order to fully consider the characteristics of the video in the subtitle, audio and visual dimensions, and no longer only consider the visual dimension. And since subsequent retrieval in this application needs to be performed through text description, the text descriptions of these three dimensions will be obtained respectively for the subtitle dimension, the audio dimension and the visual dimension, and the subsequent retrieval will be further performed using this dimension, so as to perform retrieval from these three target retrieval dimensions respectively.

[0091] Specifically, since the subtitles are originally text, directly extracting the subtitles of each segmented video can obtain the text description in the subtitle dimension. The text description of the audio can be obtained by converting the audio of the video into text. The text description in the visual dimension is the information that describes the video content through text. Optionally, it can be specifically obtained by analyzing each segmented video through a model. For example, it can be specifically analyzed through the Qwen2-VL model.

[0092] It should be noted that in order to consider the overall context information and fine-grained information of the video, and to avoid the influence of redundant and unimportant information and improve the retrieval accuracy, in the embodiments of the present application, the text description in the visual dimension of a segmented video is composed of the context text description of the segmented video and the text descriptions of the key frames of each frame of the segmented video after screening. Therefore, it is necessary to screen out the key frames from the video frames of each segmented video, and analyze the segmented video and each frame of key frames through a model respectively, so as to obtain the text description of each segmented video and the text description of the key frames, and combine the text descriptions of each segmented video and the text descriptions of the key frames to obtain the text description of the segmented video.

[0093] Optionally, in another embodiment of the present application, when the target retrieval dimension is the subtitle dimension, a specific implementation manner of step S103 includes:

[0094] Performing character recognition technology on each frame of the video frames of each segmented video to obtain a segmented subtitle content library composed of the subtitle content of each segmented video.

[0095] Specifically, using OCR technology to extract the subtitles on each frame of the video frames of each segmented video, so as to obtain the subtitle content of each segmented video, and combining the subtitle content of each segmented video to form a segmented subtitle content library.

[0096] Optionally, in another embodiment of the present application, when the target retrieval dimension is the visual dimension, a specific implementation manner of step S103 is as Figure 2 shown and includes the following steps:

[0097] S201. Using a multimodal large model to perform descriptive analysis on each segmented video to obtain the context text description of each segmented video, and using a multimodal large model to perform descriptive analysis on each frame of the video frames of each segmented video to obtain the text description of each frame of the video frames of each segmented video.

[0098] Specifically, in the embodiments of the present application, a multi-modal large model, such as Qwen2-VL, is used to describe and analyze each sub-shot video to obtain the context text description of each sub-shot video, that is, a coarse-grained visual text description is obtained. Moreover, the text description of each video frame of each sub-shot video is also obtained by using the multi-modal large model to describe and analyze each video frame of each sub-shot video.

[0099] S202. Use the pre-trained information quantity score model to score each video frame of each sub-shot video.

[0100] Different from the existing method of filtering key frames or high-information frames based on image similarity, in the embodiments of the present application, the trained information quantity score model is used to score each video frame, and then the video frames containing key information are screened out according to the scores of each video frame, the low-information video frames are filtered out, and the high-information video frames are retained to achieve further refinement of fine-grained frames.

[0101] Optionally, in another embodiment of the present application, a specific implementation manner of step S202 is as Figure 3 shown, and includes the following steps:

[0102] S301. For each video frame of each sub-shot video, input the video frame into the image encoding module in the pre-trained information quantity score model.

[0103] The information quantity score model includes an image encoding module, a text encoding module, and a loss function, and is jointly trained. The image encoding module is used to process images. The text encoding module is used to process text, specifically to process the visual text description of the image. Although the processing results of an image and its visual text description are of different types, their trends are the same. Therefore, in the embodiments of the present application, by jointly training the image encoding module and the text encoding module, the image encoding module can be effectively improved. Therefore, when scoring a video frame, it only needs to be input into the image encoding module to score it.

[0104] The loss function is used to calculate the loss of the analysis result during the training process to determine whether the model training meets the requirements.

[0105] Optionally, in the embodiments of the present application, the loss function includes two parts. One part is the margin-based MFF loss function innovatively proposed in the embodiments of the present application, and its specific form is:

[0106]

[0107] where N represents the batch size; m represents the margin, usually defaulting to 0.5; and respectively represent the embedding feature vectors of the i-th text and the corresponding image, that is, the vectors output by the corresponding image encoding module and the output of the text encoding module; represents L2 normalization; represents the weight ratio, usually defaulting to 0.7; s represents the scaling parameter, usually defaulting to 32.

[0108] Another part is the image information quantity score loss function, and specifically it is:

[0109]

[0110] where, y i represents the score corresponding to the i-th image, and y j represents the score corresponding to the j-th image, that is, the score of the image output by the image encoding module; N represents the batch size; α and β represent the scale parameters, usually set to 0.7 and 0.3 respectively.

[0111] S302. Extract spatio-temporal domain features from the video frame through the feature preprocessor in the image encoding module.

[0112] Optionally, specifically, vit can be used for image feature preprocessing, and through it, the features of the image are extracted, that is, the spatio-temporal domain features of the image are extracted. Then it is sent to the fine-grained frequency space implicit encoder module and the fine-grained separation module based on the frequency domain space for processing. It should be noted that during training, the image embedding layer also needs to be input simultaneously to obtain the corresponding feature vectors, which are combined with the feature vectors output by the text encoding module through the input vectors to realize the joint training of the two modules.

[0113] S303. Use the attention-based discrete wavelet transform method to transform the spatio-temporal domain features to the frequency domain space for attention mechanism learning, and obtain the attention frequency domain space matrix.

[0114] Since it is not easy to learn the image information quantity in the spatio-temporal domain, in the embodiment of the present application, the attention-based DWT (discrete wavelet transform) method is used to transform the spatio-temporal domain features to the frequency domain space for attention mechanism learning. The specific operation can be: for the feature maps of K channels, use K DWTs to transform the feature maps into 4*K frequency domain space matrices. Then use 4*K*3*3 convolution, and then average pooling to obtain K correlation coefficients. Then multiply with the 4*K frequency domain space matrices to obtain the attention frequency domain space matrix.

[0115] S304. Perform inverse transformation on the attention frequency domain space matrix through the inverse discrete wavelet transform method to obtain the attention-based transformation feature matrix.

[0116] Specifically, the inverse IDWT is used to perform inverse transformation on the attention frequency domain space matrix to obtain the DWT transformation features based on attention, and then they are fed into the fully connected layer (with a dimension of 1*d).

[0117] S305. The fine-grained separation module based on the frequency domain space is used to perform fine-grained separation on the transformation feature matrix based on attention to obtain the fine-grained frequency domain feature matrix.

[0118] To further separate 256 fine-grained parts, the embodiment of the present application creatively proposes a fine-grained separation module based on the frequency domain space. First, a matrix with a dimension of d*d*256 is added, and finally a d*256 feature matrix is output. Among them, 256 fine-grained spaces are sufficient to cover the fine-grained features of the entire image. Then, the DWT is used again to map the feature matrix to the frequency space according to the corresponding number of channels (256 channels) to obtain the frequency domain features (d / 2*4*256). Since the fine-grained parts are reflected in the high frequency, the low-frequency components are discarded, and then the fine-grained frequency domain features (d / 2*3*256) are obtained.

[0119] S306. Based on the fine-grained frequency domain feature matrix, the information quantity scoring layer outputs the score of the video frame.

[0120] In the embodiment of the present application, finally, an information quantity scoring layer is connected, and the score of the video frame is output through it.

[0121] Specifically, for the information quantity scoring layer, it models the implicit association by treating the features as graph nodes. Therefore, the feature similarity can be calculated using the query matrix and the key matrix to determine the score. Among them, the specific steps are as follows: first, a feature graph is constructed, then the feature graph is convolved to enhance the features, and finally, the score prediction is based on the features.

[0122] S203. Based on the scores of each video frame, each key frame of each shot video is screened out from each video frame.

[0123] Specifically, the video frames with scores higher than the score threshold can be screened out as key frames.

[0124] S204. The context text description of each shot video and the text descriptions of each key frame of the shot video are combined respectively to obtain a shot visual multi-grained text description library composed of various shot visual multi-grained text descriptions.

[0125] Specifically, the context text description of each shot video is combined with the text descriptions of its each key frame to obtain various shot visual multi-grained text descriptions, and the various shot visual multi-grained text descriptions are combined to form a shot visual multi-grained text description library.

[0126] Optionally, in another embodiment of the present application, when the target retrieval dimension is the audio dimension, a specific implementation of step S103 includes:

[0127] Extract the audio data of each sub-shot video, and use the audio language large model to extract information from the audio data of each sub-shot video to obtain a sub-shot audio information text description library composed of text descriptions of the audio information of each sub-shot.

[0128] Specifically, extract the audio data of each sub-shot video, and then convert the audio data into text information through the audio language large model for subsequent use. Optionally, specifically, the Qwen2-Audio audio language large model or other large models can be used to extract the text descriptions of the audio information of each sub-shot video, that is, the text descriptions of the audio information of each sub-shot, and the text descriptions of the audio information of each sub-shot form a sub-shot audio information text description library.

[0129] S104. Combine the text descriptions of the target retrieval dimension of each sub-shot video with the retrieval question to perform retrieval enhancement for the target retrieval dimension, and obtain multiple retrieval videos based on the target retrieval dimension.

[0130] It should be noted that after obtaining the text description of a target retrieval dimension, since the text description of the target retrieval dimension is the feature of the target retrieval dimension, the text description of the target retrieval dimension can be combined with the retrieval question input by the user for retrieval, so that video retrieval can be performed from the target retrieval dimension, and multiple retrieval videos based on the target retrieval dimension can be obtained. That is, video retrieval is performed separately from the subtitle dimension, the audio dimension, and the visual dimension, and then multiple retrieval videos based on these three dimensions are obtained.

[0131] In order to perform more accurate retrieval, the retrieval enhancement generation technology is used for video retrieval. The retrieval model is combined with the large model, and the text description of the target retrieval dimension is combined with the retrieval question, and then the large language model retrieves the accurate video.

[0132] Optionally, in another embodiment of the present application, a specific implementation of step S104 is as Figure 4 shown, including the following steps:

[0133] S401. Convert the text descriptions of the target retrieval dimension of each sub-shot video into feature vectors of the target retrieval dimension of each sub-shot video.

[0134] It should be noted that since the data volume of the text descriptions of the target retrieval dimensions of each segmented video obtained by analysis is large, and there may be text descriptions with low information, in order to improve the retrieval accuracy, the text descriptions will be screened before retrieval. And in order to perform the screening, it is necessary to convert the text descriptions of the target retrieval dimensions of each segmented video into feature vectors of the target retrieval dimensions of each segmented video for the convenience of screening based on the feature vectors. Optionally, a bert model or other models can be used to convert the text descriptions into vectors.

[0135] S402. Based on the feature vectors of the target retrieval dimensions of each segmented video and the feature vectors of the retrieval problem, screen out multiple feature vectors of the target retrieval dimensions similar to the retrieval problem.

[0136] Specifically, screening can be performed through a rough ranking semantic retrieval model. Specifically, the rough ranking semantic retrieval model uses the feature vectors of each target retrieval dimension and the feature vectors of the retrieval problem to calculate the similarity between each text description and the retrieval problem. Then, the K text descriptions with the largest similarity are screened out.

[0137] S403. Combine the text descriptions of each target retrieval dimension corresponding to the screened feature vectors of each target retrieval dimension with the prompt information and the retrieval problem, and input them into a large language model for retrieval enhancement to obtain multiple retrieval videos based on the target retrieval dimension.

[0138] It should be noted that in the embodiments of the present application, video retrieval is performed by means of retrieval enhancement through a large language model. And in order to enable the large language model to accurately generate the required time, on the basis of the text descriptions of each target retrieval dimension and the retrieval problem, prompt information is further added, so that the large language model performs retrieval according to the prompt of the prompt information.

[0139] S105. Weight the text descriptions of each target retrieval dimension of each retrieval video to obtain multiple segmented multi-dimensional text descriptions.

[0140] Since video retrieval is performed based on each target retrieval dimension through the above steps, that is, each target retrieval dimension is considered separately for retrieval, accurate retrieval can be focused on each target retrieval dimension, accurate retrieval results based on each target retrieval dimension can be obtained, and the data volume for subsequent processing is reduced. However, since only each target retrieval dimension is considered separately, in order to further improve the retrieval accuracy, therefore, on the basis of each retrieval video retrieved through three target retrieval dimensions, each target retrieval dimension is further considered comprehensively for retrieval, so the text descriptions of each target retrieval dimension of each retrieval video are weighted to obtain multiple segmented multi-dimensional text descriptions.

[0141] S106. Combine the multi-dimensional text descriptions of each sub-shot with the retrieval question for retrieval enhancement to obtain the final retrieval result.

[0142] Similarly, after obtaining the multi-dimensional text descriptions of each sub-shot, the features of the three target retrieval dimensions are integrated. Therefore, combining it with the set of retrieval questions, videos that better match the retrieval questions are further retrieved from each retrieved video to obtain the final retrieval result, thereby effectively improving the retrieval accuracy.

[0143] Optionally, in another embodiment of the present application, the specific implementation manner of step S106 includes:

[0144] Based on the feature vectors of the multi-dimensional text descriptions of each sub-shot and the feature vector of the retrieval question, multiple multi-dimensional text descriptions of the sub-shots similar to the retrieval question are screened out, and the screened multi-dimensional text descriptions of each sub-shot are combined with the prompt information and the retrieval question and input into the large language model for retrieval enhancement to obtain the final retrieval result.

[0145] It should be noted that, also for improving the retrieval accuracy, based on the feature vectors of the multi-dimensional text descriptions of each sub-shot and the feature vector of the retrieval question, the similarity between each multi-dimensional text description of the sub-shot and the construction question is calculated. Then, multiple multi-dimensional text descriptions of the sub-shots with the highest similarity are screened out, and the screened multi-dimensional text descriptions of each sub-shot are combined with the prompt information and the retrieval question and input into the large language model for retrieval enhancement to obtain the final retrieval result.

[0146] An embodiment of the present application provides a video retrieval method. It acquires each current video to be retrieved, and divides each video to be retrieved into multiple sub-shot videos respectively, so as to perform retrieval from the dimension of sub-shots and ensure the accuracy of retrieval. For each target retrieval dimension respectively, it acquires the text description of the target retrieval dimension of each sub-shot video, and combines the text description of the target retrieval dimension of each sub-shot video with the retrieval problem to perform retrieval enhancement for the target retrieval dimension, and obtains multiple retrieval videos based on the target retrieval dimension. Among them, the target retrieval dimensions include the subtitle dimension, the audio dimension, and the visual dimension, so as to perform precise video retrieval from three dimensions by extracting the text descriptions of the subtitle, audio, and visual dimensions. And, the text description of the visual dimension of a sub-shot video is composed of the context text description of the sub-shot video and the text descriptions of the key frames of each frame of the sub-shot video selected, so as to fully consider the context information of the video, also consider the fine-grained information of each video frame, and filter out the information of low-information video frames to avoid affecting the accuracy of the retrieval result, effectively ensuring the accuracy of the retrieval dimension in the visual dimension. Finally, it weights the text descriptions of each target retrieval dimension of each retrieval video to obtain multiple multi-dimensional text descriptions of sub-shots, and combines each multi-dimensional text description of sub-shots with the retrieval problem to perform retrieval enhancement to obtain the final retrieval result, so as to further fuse the features of the subtitle, audio, and visual dimensions on the basis of each retrieval video retrieved from the three target retrieval dimensions and perform further retrieval, thus effectively ensuring the accuracy of the retrieval.

[0147] Another embodiment of the present application provides a video retrieval device, as Figure 5 shown, including:

[0148] A video acquisition unit 501, configured to acquire each current video to be retrieved.

[0149] A sub-shot division unit 502, configured to divide each video to be retrieved into multiple sub-shot videos respectively.

[0150] A text description acquisition unit 503, configured to acquire the text description of the target retrieval dimension of each sub-shot video for each target retrieval dimension respectively.

[0151] Among them, the target retrieval dimensions include the subtitle dimension, the audio dimension, and the visual dimension. The text description of the visual dimension of a sub-shot video is composed of the context text description of the sub-shot video and the text descriptions of the key frames of each frame of the sub-shot video selected.

[0152] A preliminary retrieval unit 504, configured to combine the text description of the target retrieval dimension of each sub-shot video with the retrieval problem to perform retrieval enhancement for the target retrieval dimension, and obtain multiple retrieval videos based on the target retrieval dimension.

[0153] The fusion unit 505 is used to weight the text descriptions of each target retrieval dimension of each retrieved video to obtain multiple multi-dimensional text descriptions of sub-scenes.

[0154] The final retrieval unit 506 is used to combine each multi-dimensional text description of sub-scenes with the retrieval question for retrieval enhancement to obtain the final retrieval result.

[0155] Optionally, in the video retrieval device provided in another embodiment of the present application, when the target retrieval dimension is the subtitle dimension, the text description acquisition unit executes to obtain the text description of the target retrieval dimension of each sub-scene video, for:

[0156] Perform character recognition on each video frame of each sub-scene video through character recognition technology to obtain a sub-scene subtitle content library composed of the subtitle content of each sub-scene video.

[0157] Optionally, in the video retrieval device provided in another embodiment of the present application, when the target retrieval dimension is the visual dimension, the text description acquisition unit executes to obtain the text description of the target retrieval dimension of each sub-scene video, for:

[0158] Use a multi-modal large model to perform description analysis on each sub-scene video to obtain the context text description of each sub-scene video, and use the multi-modal large model to perform description analysis on each video frame of each sub-scene video to obtain the text description of each video frame of each sub-scene video.

[0159] Use a pre-trained information quantity scoring model to score each video frame of each sub-scene video.

[0160] Based on the scores of each video frame, screen out the key frames of each sub-scene video from each video frame.

[0161] Combine the context text description of each sub-scene video and the text description of the key frames of each sub-scene video respectively to obtain a sub-scene visual multi-granularity text description library composed of each sub-scene visual multi-granularity text description.

[0162] Optionally, in the video retrieval device provided in another embodiment of the present application, when the text description acquisition unit executes to use a pre-trained information quantity scoring model to score each video frame of each sub-scene video, for:

[0163] For each video frame of each sub-scene video respectively, input the video frame into the image encoding module in the pre-trained information quantity scoring model. The information quantity scoring model includes an image encoding module, a text encoding module, and a loss function, and is jointly trained.

[0164] Extract spatio-temporal domain features from the video frame through the feature pre-processor in the image encoding module.

[0165] Using the attention-based discrete wavelet transform method, the spatio-temporal domain features are transformed into the frequency domain space for attention mechanism learning to obtain an attention frequency domain space matrix.

[0166] Performing an inverse transform on the attention frequency domain space matrix through the inverse discrete wavelet transform method to obtain an attention-based transformed feature matrix.

[0167] Performing fine-grained separation on the attention-based transformed feature matrix through a fine-grained separation module based on the frequency domain space to obtain a fine-grained frequency domain feature matrix.

[0168] Outputting the score of the video frame through an information quantity scoring layer based on the fine-grained frequency domain feature matrix.

[0169] Optionally, in the video retrieval device provided in another embodiment of the present application, when the target retrieval dimension is the audio dimension, the text description acquisition unit executes to obtain the text description of the target retrieval dimension of each segmented video for:

[0170] Extracting the audio data of each segmented video.

[0171] Using an audio language large model to extract information from the audio data of each segmented video to obtain a segmented audio information text description library composed of the text descriptions of the segmented audio information of each segmented video.

[0172] Optionally, in the video retrieval device provided in another embodiment of the present application, the preliminary retrieval unit includes:

[0173] A first conversion unit for converting the text description of the target retrieval dimension of each segmented video into a feature vector of the target retrieval dimension of each segmented video.

[0174] A first screening unit for screening out multiple feature vectors of the target retrieval dimension similar to the retrieval problem based on the feature vectors of the target retrieval dimension of each segmented video and the feature vector of the retrieval problem.

[0175] A first retrieval unit for combining the text descriptions of the respective target retrieval dimensions corresponding to the screened feature vectors of the respective target retrieval dimensions with the prompt information and the retrieval problem, and inputting them into a large language model for retrieval enhancement to obtain multiple retrieval videos based on the target retrieval dimension.

[0176] Optionally, in the video retrieval device provided in another embodiment of the present application, the final retrieval unit includes:

[0177] A second screening unit for screening out multiple segmented multi-dimensional text descriptions similar to the retrieval problem based on the feature vectors of the segmented multi-dimensional text descriptions of each and the feature vector of the retrieval problem.

[0178] A second retrieval unit, configured to combine the screened multi-dimensional text descriptions of each sub-shot with the prompt information and the retrieval question, and input the combination into a large language model for retrieval enhancement to obtain a final retrieval result.

[0179] It should be noted that for the specific working processes of the various units provided in the above embodiments of the present application, reference may be made correspondingly to the implementation processes of the corresponding steps in the above method embodiments, which will not be elaborated herein.

[0180] Another embodiment of the present application provides an electronic device, as Figure 6 shown, including:

[0181] A memory 601 and a processor 602.

[0182] Among them, the memory 601 is used to store programs.

[0183] The processor 602 is used to execute the programs stored in the memory 601. When the programs are executed, they are specifically used to implement the video retrieval method provided in any of the above embodiments.

[0184] Another embodiment of the present application provides a computer storage medium for storing a computer program, which is used to implement the video retrieval method provided in any of the above embodiments when executed by a processor.

[0185] Computer storage media include both permanent and non-permanent, removable and non-removable media and can be implemented by any method or technology for storing information. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transitory media that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0186] Those skilled in the art may further realize that the units and algorithm steps of each example described in connection with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of this application.

[0187] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to these embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video retrieval method, characterized in that, Including: Obtain each video to be retrieved currently; Divide each of the videos to be retrieved into multiple sub-shot videos respectively; For each target retrieval dimension respectively, obtain the text description of the target retrieval dimension of each of the sub-shot videos; wherein, the target retrieval dimension includes a subtitle dimension, an audio dimension, and a visual dimension; the text description of the visual dimension of a sub-shot video is composed of the context text description of the sub-shot video and the text descriptions of each frame of key frames of the sub-shot video screened out; Combine the text descriptions of the target retrieval dimension of each of the sub-shot videos with the retrieval question, perform retrieval enhancement for the target retrieval dimension, and obtain multiple retrieval videos based on the target retrieval dimension; Weight the text descriptions of each target retrieval dimension of each of the retrieval videos to obtain multiple multi-dimensional text descriptions of sub-shots; Combine the multi-dimensional text descriptions of each sub-shot with the retrieval question to perform retrieval enhancement and obtain the final retrieval result.

2. The method according to claim 1, wherein When the target retrieval dimension is the subtitle dimension, obtaining the text description of the target retrieval dimension of each of the sub-shot videos includes: Perform character recognition on each video frame of each of the sub-shot videos through character recognition technology to obtain a sub-shot subtitle content library composed of the subtitle content of each of the sub-shot videos.

3. The method according to claim 1, characterized in that When the target retrieval dimension is the visual dimension, obtaining the text description of the target retrieval dimension of each of the sub-shot videos includes: Use a multi-modal large model to perform descriptive analysis on each of the sub-shot videos to obtain the context text description of each of the sub-shot videos, and use the multi-modal large model to perform descriptive analysis on each video frame of each of the sub-shot videos to obtain the text description of each video frame of each of the sub-shot videos; Use a pre-trained information quantity scoring model to score each video frame of each of the sub-shot videos; Based on the scores of each video frame, screen out each frame of key frames of each of the sub-shot videos from each video frame; Combine the context text description of each sub-shot video and the text descriptions of each frame of key frames of the sub-shot video respectively to obtain a sub-shot visual multi-granularity text description library composed of multi-granularity text descriptions of each sub-shot visual.

4. The method according to claim 1, characterized in that The using a pre-trained information quantity scoring model to score each video frame of each of the sub-shot videos includes: For each video frame of each of the sub-shot videos respectively, input the video frame into the image encoding module in the pre-trained information quantity scoring model; wherein, the information quantity scoring model includes an image encoding module, a text encoding module, and a loss function, and is jointly trained; Extract spatio-temporal domain features from the video frame through the feature pre-processor in the image encoding module; Use the attention-based discrete wavelet transform method to transform the spatio-temporal domain features into the frequency domain space for attention mechanism learning to obtain an attention frequency domain space matrix; Perform inverse transform on the attention frequency domain space matrix through the inverse discrete wavelet transform method to obtain an attention-based transform feature matrix; Perform fine-grained separation on the attention-based transformed feature matrix through a fine-grained separation module based on the frequency domain space to obtain a fine-grained frequency domain feature matrix; Output the score of the video frame through an information amount scoring layer based on the fine-grained frequency domain feature matrix.

5. The method according to claim 1, characterized in that The target retrieval dimension is the audio dimension. The obtaining of the text descriptions of the target retrieval dimension of each of the sub-shot videos includes: Extract the audio data of each of the sub-shot videos; Use an audio language large model to extract information from the audio data of each of the sub-shot videos to obtain a sub-shot audio information text description library composed of text descriptions of sub-shot audio information.

6. The method according to claim 1, wherein The combining of the text descriptions of the target retrieval dimension of each of the sub-shot videos with the retrieval question to perform retrieval enhancement in the target retrieval dimension to obtain multiple retrieval videos based on the target retrieval dimension includes: Convert the text descriptions of the target retrieval dimension of each of the sub-shot videos into feature vectors of the target retrieval dimension of each of the sub-shot videos; Based on the feature vectors of the target retrieval dimension of each of the sub-shot videos and the feature vector of the retrieval question, screen out multiple feature vectors of the target retrieval dimension that are similar to the retrieval question; Combine the text descriptions of the target retrieval dimension corresponding to the screened feature vectors of the target retrieval dimension with the prompt information and the retrieval question, and input them into a large language model for retrieval enhancement to obtain multiple retrieval videos based on the target retrieval dimension.

7. The method according to claim 1, characterized in that The combining of the multi-dimensional text descriptions of each of the sub-shot videos with the retrieval question to perform retrieval enhancement to obtain the final retrieval result includes: Based on the feature vectors of the multi-dimensional text descriptions of each of the sub-shot videos and the feature vector of the retrieval question, screen out multiple multi-dimensional text descriptions of the sub-shot videos that are similar to the retrieval question; Combine the screened multi-dimensional text descriptions of each of the sub-shot videos with the prompt information and the retrieval question, and input them into a large language model for retrieval enhancement to obtain the final retrieval result.

8. A video retrieval device, characterized in that, Includes: A video acquisition unit for acquiring each of the current videos to be retrieved; A sub-shot division unit for dividing each of the videos to be retrieved into multiple sub-shot videos respectively; A text description acquisition unit for respectively acquiring the text descriptions of the target retrieval dimension of each of the sub-shot videos for each target retrieval dimension; wherein, the target retrieval dimensions include a caption dimension, an audio dimension, and a visual dimension; the text description of the visual dimension of a sub-shot video is composed of the context text description of the sub-shot video and the text descriptions of the key frames of each frame of the sub-shot video screened out; A preliminary retrieval unit for combining the text descriptions of the target retrieval dimension of each of the sub-shot videos with the retrieval question to perform retrieval enhancement in the target retrieval dimension to obtain multiple retrieval videos based on the target retrieval dimension; A fusion unit for weighting the text descriptions of the target retrieval dimension of each of the retrieval videos to obtain multiple multi-dimensional text descriptions of sub-shots; A final retrieval unit for combining each of the multi-dimensional text descriptions of the sub-scenes with the retrieval question for retrieval enhancement to obtain a final retrieval result.

9. An electronic device, characterized in that, It includes: A memory and a processor; Wherein, the memory is used for storing programs; The processor is used for executing the program, and when the program is executed, it is specifically used to implement the video retrieval method according to any one of claims 1 to 7.

10. A computer storage medium, characterized in that, For storing a computer program, when the computer program is executed by a processor, it is used to implement the video retrieval method according to any one of claims 1 to 7.

Images