Video search methods, devices, equipment, and storage media based on cross-modal retrieval

Abstract

This invention provides a video search method, apparatus, device, and storage medium based on cross-modal retrieval, belonging to the field of data processing. The method includes: acquiring retrieval text data and converting it into background and foreground images; searching for each video to be searched based on the background images to obtain a first video search result; searching for each video to be searched based on the foreground images to obtain a second video search result; and determining the video search result corresponding to the retrieval text data based on the first and second video search results. This invention can accurately represent the background and target object of the retrieval text data, shorten retrieval time, and improve retrieval accuracy.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a video search method, apparatus, device, and storage medium based on cross-modal retrieval. Background Technology

[0002] With the rapid development of internet technology, video data is growing explosively. How to quickly and accurately retrieve the videos users need from this massive amount of video data has become a pressing problem. Cross-modal video retrieval technology aims to achieve correlated retrieval between different modalities of data (such as text and video), meeting diverse user retrieval needs.

[0003] However, traditional cross-modal video retrieval generally suffers from limitations in feature extraction, failing to fully explore the deep semantic information of both text and video when extracting features. For text, it often focuses only on surface lexical features, ignoring semantic contextual relationships and complex semantic structures; for video, it extracts only simple visual features, such as color and texture, making it difficult to capture the complex events, behaviors, and scene semantics within the video. This results in the inability to accurately measure the semantic similarity between text and video during retrieval, leading to low retrieval accuracy. For example, when retrieving videos related to "an intense football match," traditional methods may return irrelevant or low-relevance videos because they cannot accurately understand the semantics of "intense" and "football match," as well as the corresponding scene and action features in the video. Moreover, as the volume of video data continues to increase, traditional retrieval methods become slow when processing large-scale data, failing to meet real-time requirements. Summary of the Invention

[0004] This invention provides a video search method, apparatus, device, and storage medium based on cross-modal retrieval to solve the problems of low retrieval accuracy and poor real-time performance in cross-modal video retrieval.

[0005] In a first aspect, embodiments of the present invention provide a video search method based on cross-modal retrieval, including:

[0006] Acquire the searched text data and convert it into a background image and a foreground image;

[0007] The first video search result is obtained by searching each video based on the background image.

[0008] The foreground image is used to search each video to be searched, and a second video search result is obtained.

[0009] Based on the first video search result and the second video search result, determine the video search result corresponding to the retrieved text data.

[0010] In one possible implementation, converting the retrieved text data into a background image and a foreground image includes:

[0011] Semantic analysis of the retrieved text data is performed based on natural language processing technology to obtain key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data.

[0012] Foreground image prompts are constructed based on the key objects and the core actions, and background image prompts are constructed based on the environmental information and the atmosphere information;

[0013] The foreground image prompt is input into the text image diffusion model to obtain the foreground image;

[0014] The background image prompt is input into the text image diffusion model to obtain the background image.

[0015] In one possible implementation, before retrieving each video to be retrieved based on the background image and obtaining the first video search result, the method further includes:

[0016] Obtain the target background image after filtering the background image;

[0017] Based on the background image, each video to be searched is retrieved to obtain the first video search result, including:

[0018] The first video search result is obtained by searching each video based on the target background image.

[0019] Before retrieving each video to be searched based on the foreground image to obtain the second video search result, the process also includes:

[0020] Obtain the target foreground image after filtering the foreground image;

[0021] Based on the foreground image, each video to be searched is retrieved to obtain a second video search result, including:

[0022] Based on the target foreground image, each video to be searched is retrieved to obtain the second video search result.

[0023] In one possible implementation, each video to be retrieved is searched based on the target background image to obtain a first video search result, including:

[0024] For each video to be searched, extract each keyframe from the video and construct a background image for the video based on each keyframe, which is denoted as a candidate background image.

[0025] Calculate the similarity between the target background image and the candidate background images, and select the video segments corresponding to the candidate background images with a similarity greater than a preset background threshold as the first video search results.

[0026] In one possible implementation, determining the video search results corresponding to the retrieved text data based on the first video search result and the second video search result includes:

[0027] A second search is performed on each of the first video search results based on the foreground image to obtain the third video search result;

[0028] Based on the background image, a second search is performed on each of the second video search results to obtain the fourth video search result;

[0029] The union of the third video search result and the fourth video search result is used to obtain the video search result corresponding to the search text data.

[0030] In one possible implementation, the video search method based on cross-modal retrieval further includes:

[0031] The retrieved text data is encoded using a text encoder to obtain a text feature vector;

[0032] Each video to be retrieved is encoded using a video encoder to obtain the video feature vector corresponding to each video to be retrieved.

[0033] Based on the similarity between the text feature vector and each of the video feature vectors, candidate videos are determined from each video to be retrieved;

[0034] After determining the video search results corresponding to the retrieved text data based on the first video search results and the second video search results, the method further includes:

[0035] Find the intersection of the candidate videos and the video search results, and determine the final video search results corresponding to the search text data based on the intersection.

[0036] Secondly, embodiments of the present invention provide a video search device based on cross-modal retrieval, comprising:

[0037] The processing module is used to acquire the retrieved text data and convert the retrieved text data into a background image and a foreground image;

[0038] The first retrieval module is used to retrieve each video to be retrieved based on the background image and obtain the first video search result;

[0039] The second retrieval module is used to retrieve each video to be retrieved based on the foreground image, and obtain the second video search results;

[0040] The search result determination module is used to determine the video search results corresponding to the searched text data based on the first video search results and the second video search results.

[0041] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect or any possible implementation thereof.

[0042] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0043] Fifthly, embodiments of the present invention provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect or any possible implementation thereof.

[0044] This invention provides a video search method, apparatus, device, and storage medium based on cross-modal retrieval. After acquiring retrieval text data, the retrieval text data is converted into background and foreground images. Then, each video to be retrieved is searched based on the background image to obtain a first video search result. Each video to be retrieved is then searched based on the foreground image to obtain a second video search result. Finally, based on the first and second video search results, the video search result corresponding to the retrieval text data is determined. This eliminates the step of extracting video features from each video to be retrieved, which is beneficial for improving the real-time performance of video retrieval. Moreover, compared to extracting video features from each video to be retrieved, converting the retrieval text data into images is simpler and can further improve the real-time performance of retrieval. Furthermore, converting the retrieval text data into background and foreground images allows for a more accurate representation of the background and target object of the retrieval text data, thereby obtaining more accurate video search results based on the background and foreground images. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the implementation of the video search method based on cross-modal retrieval provided in this embodiment of the invention;

[0047] Figure 2 This is a schematic diagram of the structure of a video search device based on cross-modal retrieval provided in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, specific embodiments will be described below in conjunction with the accompanying drawings.

[0051] Figure 1 The following is a flowchart illustrating the implementation of the video search method based on cross-modal retrieval provided in this embodiment of the invention. Figure 1 As shown, the method includes:

[0052] Step 101: Obtain the retrieved text data and convert it into a background image and a foreground image.

[0053] In this embodiment, obtaining the retrieval text data can be understood as text data used to retrieve videos, and the content described in the retrieved videos is related to the content described in the retrieval text data.

[0054] In one embodiment, converting retrieved text data into background and foreground images may include:

[0055] Semantic analysis of retrieved text data is performed using natural language processing technology to obtain key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data.

[0056] Foreground image cues are constructed based on key objects and core actions, while background image cues are constructed based on environmental and atmospheric information.

[0057] The foreground image cues are input into the text image diffusion model to obtain the foreground image.

[0058] The background image prompt is input into the text image diffusion model to obtain the background image.

[0059] For example, natural language processing technology can be used to perform entity recognition on the retrieved text data to extract key objects such as girls, boys, cats, and dogs.

[0060] For example, natural language processing technology can be used to perform action recognition on the retrieved text data to extract the core actions in the retrieved text data, such as walking, running, riding a bicycle, dribbling a ball, etc.

[0061] For example, natural language processing technology can be used to perform scene recognition on the retrieved text data and extract environmental information from the retrieved text data, such as being in a park or at an intersection.

[0062] For example, natural language processing techniques can be used to perform sentiment and style analysis on retrieved text data to extract the emotional attributes and overall atmosphere of the retrieved text data, such as happy, intense, dreamy, realistic, etc.

[0063] After obtaining the key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data, images cannot be directly generated because text often omits a large number of visual details (such as exact color, texture, perspective, lighting, etc.). Therefore, image descriptions are constructed through cue word engineering, which structures the key objects, core actions, environmental information, and atmosphere information obtained from semantic analysis and converts them into a very detailed and keyword-rich "cue word". Images are then generated based on the "cue word".

[0064] To more accurately represent the retrieved text data during the image conversion process, the key objects, core actions, environmental information, and atmosphere information extracted from the retrieved text data are divided into two categories. The key objects and core actions represent the target objects of the retrieved text data, thus generating the foreground image. The environmental information and atmosphere information represent the background of the retrieved text data, thus generating the background image.

[0065] For example, in the process of generating images based on "cue words", the cue words can be input into text-to-image diffusion models, such as Stable Diffusion, DALL-E 3, Midjourney, etc. These models are trained on massive "text-image" pairs and can associate text descriptions with visual features, thereby "generating" a brand new image that matches the description from noise based on the semantic information in the cue words.

[0066] It should be noted that during the process of inputting foreground image cues into the text-based image diffusion model to obtain foreground images, and inputting background image cues into the text-based image diffusion model to obtain background images, multiple foreground images and multiple background images can be generated for filtering. This allows for more accurate video retrieval based on the filtered target foreground and target background images.

[0067] For example, after converting the retrieved text data into foreground and background images, they can be displayed, and users can manually select the foreground and background images to be displayed. If a foreground and background image are manually selected, a target foreground and background image that more accurately represents the search intent can be obtained. If no suitable foreground and background images are found after manual selection, they can be regenerated to more accurately express the search intent.

[0068] Step 102: Search for each video to be searched based on the background image to obtain the first video search result.

[0069] For example, before searching each video to be searched based on the background image and obtaining the first video search result, the process may further include: obtaining the target background image after filtering the background image.

[0070] Accordingly, each video to be searched is searched based on the background image to obtain the first video search result. That is, each video to be searched is searched based on the target background image to obtain the first video search result.

[0071] In one embodiment, retrieving each video to be searched based on the target background image to obtain a first video search result may include:

[0072] For each video to be searched, extract each keyframe from the video and construct a background image for the video based on each keyframe, which is denoted as a candidate background image.

[0073] Calculate the similarity between the target background image and the candidate background images, and select the video segments corresponding to the candidate background images with a similarity greater than a preset background threshold as the first video search result.

[0074] In this embodiment, each video to be retrieved can correspond to one candidate background image or multiple candidate background images. When a video to be retrieved corresponds to one candidate background image, the video segment is the video to be retrieved corresponding to the candidate background image. When a video to be retrieved corresponds to multiple candidate background images, the video to be retrieved can be divided into multiple sub-videos based on the candidate background images, and the video segment is the sub-video corresponding to the candidate background image.

[0075] In one embodiment, in order to more accurately measure the similarity between the target background image and the candidate background image, when the similarity between the target background image and the candidate background image is greater than the basic background threshold, the target background image can be corrected based on the candidate background image. For example, the environmental information of the target background image can be replaced based on the environmental information of the candidate background image, or the atmosphere information of the target background image can be replaced based on the atmosphere information of the candidate background image. Then, the similarity between the corrected background image and the candidate background image is calculated.

[0076] Step 103: Search each video to be searched based on the foreground image to obtain the second video search result.

[0077] For example, before searching each video to be searched based on the foreground image and obtaining the second video search result, the method may further include: obtaining the target foreground image after filtering the foreground image.

[0078] Accordingly, each video to be searched is searched based on the foreground image to obtain the second video search result. That is, each video to be searched is searched based on the target foreground image to obtain the second video search result.

[0079] In one embodiment, retrieving each video to be retrieved based on the foreground image to obtain a second video search result may include:

[0080] For each video to be retrieved, extract each keyframe from the video and construct a foreground image based on each keyframe, which is denoted as a candidate foreground image.

[0081] Calculate the similarity between the target foreground image and the candidate foreground images, and select the video segments corresponding to the candidate foreground images with a similarity greater than a preset foreground threshold as the second video search results.

[0082] In this embodiment, each video to be retrieved can typically correspond to multiple candidate foreground images. The video to be retrieved can be divided into multiple sub-videos based on the multiple candidate foreground images. Here, the video segments corresponding to the candidate foreground images with a similarity greater than a preset foreground threshold are selected as the second video search results, that is, the sub-videos corresponding to the candidate foreground images with a similarity greater than a preset foreground threshold are selected as the second video search results.

[0083] In one embodiment, in order to more accurately measure the similarity between the target foreground image and the candidate foreground image, when the similarity between the target foreground image and the candidate foreground image is greater than the basic foreground threshold, the target foreground image can be corrected based on the candidate foreground image. For example, the target foreground image can be corrected based on the key objects or core actions of the candidate foreground image, and then the similarity between the corrected foreground image and the candidate foreground image can be calculated.

[0084] Step 104: Determine the video search results corresponding to the retrieved text data based on the first video search results and the second video search results.

[0085] In one embodiment, determining the video search results corresponding to the retrieved text data based on the first video search results and the second video search results may include:

[0086] A secondary search is performed on each of the first video search results based on the foreground image to obtain the third video search results.

[0087] A second search is performed on each of the second video search results based on the background image to obtain the fourth video search result.

[0088] Find the union of the third and fourth video search results to obtain the video search results corresponding to the retrieved text data.

[0089] In this embodiment, considering that the first video search result can represent the video search result corresponding to the background of the retrieved text data, and the second video search result can represent the video search result corresponding to the target object of the retrieved text data, and since the video search result is a combination of the background and the target object, a second search can be performed on each of the first video search results based on the foreground image to obtain a third video search result. This third video search result can represent the video search result corresponding to both the background and the target object of the retrieved text data. Then, a second search can be performed on each of the second video search results based on the background image to obtain a fourth video search result. This fourth video search result can also represent the video search result corresponding to both the background and the target object of the retrieved text data. Thus, by mutually verifying the third and fourth video search results, and utilizing the union of the third and fourth video search results, a more accurate and comprehensive video search result for the retrieved text data can be obtained.

[0090] This invention, in its embodiments, acquires retrieval text data and then converts it into background and foreground images. It then searches for each video to be retrieved based on the background image to obtain a first video search result; it searches for each video to be retrieved based on the foreground image to obtain a second video search result; finally, based on the first and second video search results, it determines the video search result corresponding to the retrieval text data. This eliminates the step of extracting video features from each video to be retrieved, improving the real-time performance of video retrieval. Furthermore, converting the retrieval text data into images is simpler than extracting video features from each video to be retrieved, further improving real-time performance. In addition, converting the retrieval text data into background and foreground images allows for a more accurate representation of the background and target object of the retrieval text data, resulting in more precise video search results based on the background and foreground images.

[0091] In another embodiment, the video search method based on cross-modal retrieval may further include:

[0092] The retrieved text data is encoded using a text encoder to obtain text feature vectors.

[0093] Each video to be retrieved is encoded using a video encoder to obtain the video feature vector corresponding to each video to be retrieved.

[0094] Candidate videos are determined from each video to be retrieved based on the similarity between the text feature vector and the feature vectors of each video.

[0095] Based on this, after determining the video search results corresponding to the retrieved text data according to the first video search results and the second video search results, it may also include:

[0096] Find the intersection of candidate videos and video search results, and determine the final video search results corresponding to the retrieved text data based on this intersection.

[0097] In this embodiment, for scenarios requiring high accuracy in video retrieval, the results of extracting text feature vectors and video feature vectors separately using dual encoders can be combined. Videos determined based on the similarity between text feature vectors and various video feature vectors are used as candidate videos. By finding the intersection of candidate videos and video search results, the retrieval results, which compress the video to be retrieved and the retrieved text data into a vector in the latent space, as well as the retrieval results that convert the retrieved text data into images, are utilized. This fully mines information from both textual and visual modalities to further improve retrieval accuracy.

[0098] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0099] The following are device embodiments of the present invention. For details not described in detail, please refer to the corresponding method embodiments described above.

[0100] Figure 2 A schematic diagram of the structure of a video search device based on cross-modal retrieval provided in an embodiment of the present invention is shown. For ease of explanation, only the parts related to the embodiment of the present invention are shown, and are described in detail below:

[0101] like Figure 2 As shown, the video search device based on cross-modal retrieval includes:

[0102] Processing module 21 is used to acquire the retrieved text data and convert the retrieved text data into background and foreground images.

[0103] The first search module 22 is used to search for each video to be searched based on the background image and obtain the first video search result.

[0104] The second search module 23 is used to search for each video to be searched based on the foreground image and obtain the second video search results.

[0105] The retrieval result determination module 24 is used to determine the video search results corresponding to the retrieval text data based on the first video search result and the second video search result.

[0106] In one possible implementation, processing module 21 is specifically used for:

[0107] Semantic analysis of retrieved text data is performed using natural language processing technology to obtain key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data.

[0108] Foreground image cues are constructed based on key objects and core actions, while background image cues are constructed based on environmental and atmospheric information.

[0109] The foreground image cues are input into the text image diffusion model to obtain the foreground image.

[0110] The background image prompt is input into the text image diffusion model to obtain the background image.

[0111] In one possible implementation, the processing module 21 is further configured to:

[0112] Retrieve the target background image after filtering the background images.

[0113] The first retrieval module 22 is specifically used for:

[0114] The search is performed on each video based on the target background image to obtain the first video search result.

[0115] Processing module 21 is also used for:

[0116] Obtain the target foreground image after filtering the foreground images.

[0117] The second retrieval module 23 is specifically used for:

[0118] The search is performed on each video to be searched based on the target foreground image, and the second video search result is obtained.

[0119] In one possible implementation, the first retrieval module 22 is specifically used for:

[0120] For each video to be searched, extract each keyframe from the video and construct a background image for the video based on each keyframe, which is denoted as a candidate background image.

[0121] Calculate the similarity between the target background image and the candidate background images, and select the video segments corresponding to the candidate background images with a similarity greater than a preset background threshold as the first video search result.

[0122] In one possible implementation, the retrieval result determination module 24 is specifically used for:

[0123] A secondary search is performed on each of the first video search results based on the foreground image to obtain the third video search results.

[0124] A second search is performed on each of the second video search results based on the background image to obtain the fourth video search result.

[0125] Find the union of the third and fourth video search results to obtain the video search results corresponding to the retrieved text data.

[0126] In one possible implementation, the retrieval result determination module 24 is further configured to:

[0127] The retrieved text data is encoded using a text encoder to obtain text feature vectors.

[0128] Each video to be retrieved is encoded using a video encoder to obtain the video feature vector corresponding to each video to be retrieved.

[0129] Candidate videos are determined from each video to be retrieved based on the similarity between the text feature vector and the feature vectors of each video.

[0130] Find the intersection of candidate videos and video search results, and determine the final video search results corresponding to the retrieved text data based on this intersection.

[0131] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Figure 3 As shown, the electronic device 3 of this embodiment includes a processor 30 and a memory 31. The memory 31 stores a computer program 32. When the processor 30 executes the computer program 32, it implements the steps in the various method embodiments described above. Alternatively, when the processor 30 executes the computer program 32, it implements the functions of each module / unit in the various device embodiments described above.

[0132] For example, computer program 32 may be divided into one or more modules / units, which are stored in memory 31 and executed by processor 30 to complete the present invention. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in electronic device 3.

[0133] Electronic device 3 may include, but is not limited to, processor 30 and memory 31. Those skilled in the art will understand that... Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 3 may also include input / output devices, network access devices, buses, etc.

[0134] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0135] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 3. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store the computer program 32 and other programs and data required by the electronic device 3. The memory 31 can also be used to temporarily store data that has been output or will be output.

[0136] For the sake of simplicity and clarity, only the above-described functional modules / units are used as examples. In practical applications, the functions described above can be assigned to different functional modules / units as needed. These modules / units can be implemented in hardware, software, or a combination of both.

[0137] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0138] This invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the methods described in the above-described method embodiments.

[0139] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0140] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. Unless otherwise specified or in conflict with logic, the terminology and / or descriptions between different embodiments are consistent and can be referenced interchangeably. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0141] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video search method based on cross-modal retrieval, characterized in that, include: Acquire the searched text data and convert it into a background image and a foreground image; The first video search result is obtained by searching each video based on the background image. The foreground image is used to search for each video to be searched, and a second video search result is obtained. Based on the first video search result and the second video search result, determine the video search result corresponding to the retrieved text data; The step of converting the retrieved text data into background and foreground images includes: Semantic analysis of the retrieved text data is performed based on natural language processing technology to obtain key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data. Construct foreground image prompts based on the key objects and core actions, and construct background image prompts based on the environmental information and atmosphere information; The foreground image prompt is input into the text image diffusion model to obtain the foreground image; The background image prompt is input into the text image diffusion model to obtain the background image; Determining the video search results corresponding to the retrieved text data based on the first video search result and the second video search result includes: A second search is performed on each of the first video search results based on the foreground image to obtain the third video search result; Based on the background image, a second search is performed on each of the second video search results to obtain the fourth video search result; The union of the third video search result and the fourth video search result is used to obtain the video search result corresponding to the search text data.

2. The video search method based on cross-modal retrieval according to claim 1, characterized in that, Before retrieving each video to be searched based on the background image and obtaining the first video search result, the process also includes: Obtain the target background image after filtering the background image; Based on the background image, each video to be searched is retrieved to obtain the first video search result, including: The first video search result is obtained by searching each video based on the target background image. Before retrieving each video to be searched based on the foreground image to obtain the second video search result, the process also includes: Obtain the target foreground image after filtering the foreground image; Based on the foreground image, each video to be searched is retrieved to obtain a second video search result, including: Based on the target foreground image, each video to be searched is retrieved to obtain the second video search result.

3. The video search method based on cross-modal retrieval according to claim 2, characterized in that, Based on the target background image, each video to be searched is retrieved to obtain the first video search result, including: For each video to be searched, extract each keyframe from the video and construct a background image for the video based on each keyframe, which is denoted as a candidate background image. Calculate the similarity between the target background image and the candidate background images, and select the video segments corresponding to the candidate background images with a similarity greater than a preset background threshold as the first video search results.

4. The video search method based on cross-modal retrieval according to claim 1, characterized in that, Also includes: The retrieved text data is encoded using a text encoder to obtain a text feature vector; Each video to be retrieved is encoded using a video encoder to obtain the video feature vector corresponding to each video to be retrieved. Based on the similarity between the text feature vector and each of the video feature vectors, candidate videos are determined from each video to be retrieved; After determining the video search results corresponding to the retrieved text data based on the first video search results and the second video search results, the method further includes: Find the intersection of the candidate videos and the video search results, and determine the final video search results corresponding to the search text data based on the intersection.

5. A video search device based on cross-modal retrieval, characterized in that, include: The processing module is used to acquire the retrieved text data and convert the retrieved text data into a background image and a foreground image; The first retrieval module is used to retrieve each video to be retrieved based on the background image and obtain the first video search result; The second retrieval module is used to retrieve each video to be retrieved based on the foreground image, and obtain the second video search results; The retrieval result determination module is used to determine the video search results corresponding to the retrieval text data based on the first video search results and the second video search results; The processing module is specifically used for: Semantic analysis of the retrieved text data is performed based on natural language processing technology to obtain key objects, core actions, environmental information, and atmosphere information corresponding to the retrieved text data. Construct foreground image prompts based on the key objects and core actions, and construct background image prompts based on the environmental information and atmosphere information; The foreground image prompt is input into the text image diffusion model to obtain the foreground image; The background image prompt is input into the text image diffusion model to obtain the background image; The search result determination module is specifically used for: A second search is performed on each of the first video search results based on the foreground image to obtain the third video search result; Based on the background image, a second search is performed on each of the second video search results to obtain the fourth video search result; The union of the third video search result and the fourth video search result is used to obtain the video search result corresponding to the search text data.

6. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

8. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 4.

Images