A content search method and related device

Abstract

This application discloses a content search method and related equipment. The related equipment may include a content search device, an electronic device, a computer program product, and a computer-readable storage medium. After obtaining a content search request, this application selects at least one candidate content from a preset content set based on at least one search suggestion carried in the content search request, extracts modal information of at least one modality from the candidate content, generates target search text corresponding to the content search request based on the modal information, retrieves at least one auxiliary search image based on the search suggestion, generates a target search image corresponding to the content search request based on the target search text and the auxiliary search image, and determines the target search content corresponding to the target search request based on the target search text and the target search image. The target search content includes content of at least one modality. This solution can improve the accuracy of content search.

Description

Technical Field

[0001] This invention relates to the field of content search, and more specifically to a content search method and related equipment, which may include content search devices, electronic devices, computer program products, and computer-readable storage media. Background Technology

[0002] In recent years, with the rapid development of internet technology, content search has become increasingly convenient. To improve the accuracy of content, multimodal data is often used for searching. Current content search methods can use multimodal data to generate text search content and embed relevant images within the text search content to obtain the target search content.

[0003] In the process of researching and practicing current technologies, the inventors of this application discovered that there may be a mismatch between the generated text and the embedded image. In addition, the embedded image may often contain illusions, which leads to low accuracy in content search. Summary of the Invention

[0004] This application provides a content search method and related equipment. The related equipment may include a content search device, an electronic device, a computer program product, and a computer-readable storage medium, which can improve the accuracy of content search.

[0005] A content search method, comprising:

[0006] Obtain a content search request, wherein the content search request carries at least one search suggestion;

[0007] Based on the search suggestions, at least one candidate content is selected from the preset content set, and modal information of at least one modality is extracted from the candidate content.

[0008] Based on the modal information, the target search text corresponding to the content search request is generated, and at least one auxiliary search image is retrieved according to the search prompt content;

[0009] Based on the target search text and the auxiliary search image, generate the target search image corresponding to the content search request;

[0010] Based on the target search text and the target search image, the target search content corresponding to the target search request is determined, and the target search content includes content in at least one modality.

[0011] Accordingly, embodiments of this application provide a content search device, including:

[0012] An acquisition unit is used to acquire a content search request, wherein the content search request carries at least one search suggestion;

[0013] The filtering unit is used to filter at least one candidate content from a preset content set based on the search prompt content, and extract modal information of at least one modality from the candidate content.

[0014] The retrieval unit is configured to generate target search text corresponding to the content search request based on the modal information, and retrieve at least one auxiliary search image according to the search prompt content;

[0015] The generation unit is used to generate a target search image corresponding to the content search request based on the target search text and the auxiliary search image;

[0016] The determining unit is configured to determine the target search content corresponding to the target search request based on the target search text and the target search image, wherein the target search content includes content in at least one modality.

[0017] In some embodiments, the filtering unit may be specifically used to identify the search intent corresponding to the content search request in the search prompt content, and obtain search intent information; based on the search intent information, filter at least one preset content in a preset content set to obtain initial candidate content; and segment at least one content fragment from the initial candidate content to obtain at least one candidate content.

[0018] In some embodiments, the filtering unit may be specifically used to filter at least one initial content corresponding to the search intent information from a preset content set; obtain historical search information of the target object corresponding to the content search request, and determine the content relevance parameter between the initial content and the content search request based on the historical search information; and filter at least one preset content from the initial content according to the content relevance parameter to obtain initial candidate content.

[0019] In some embodiments, the filtering unit may be specifically used to slice the initial candidate content to obtain at least one candidate content fragment; extract features from the candidate content fragments, and perform semantic analysis on the candidate content fragments based on the extracted content features to obtain semantic information of the candidate content fragments; and filter at least one candidate content fragment from the candidate content fragments based on the semantic information to obtain candidate content.

[0020] In some embodiments, the retrieval unit may be specifically used to classify the content search request according to search requirements based on the search intent information to obtain the search requirement type of the content search request; when the search requirement type is a step list requirement, generate the target search text corresponding to the content search request based on the modal information; when the search requirement type is a long text search requirement or a short text search requirement, generate the target search text corresponding to the content search request based on the search prompt content and modal information.

[0021] In some embodiments, the generation unit may be specifically used to determine the search condition information corresponding to the content search request based on the search demand type and the auxiliary search image; and based on the search condition information, to generate at least one image corresponding to the target search text using an image generation model to obtain the target search image.

[0022] In some embodiments, the generation unit may be specifically configured to: when the search request type is the step list request, identify the guiding text for each step in the target search text, and determine the search condition information corresponding to the content search request based on the guiding text, auxiliary search image, and candidate content; when the search request type is the short text search request, determine the search condition information corresponding to the content search request based on the target search text, auxiliary search image, and candidate content; and when the search request type is the long text search request, determine the search condition information corresponding to the content search request based on the candidate content and auxiliary search image.

[0023] In some embodiments, the generating unit may be specifically configured to, when the candidate content contains video content or image content, filter at least one image corresponding to the guiding text from the candidate content and auxiliary search images to obtain a search condition image, and use the search condition image and the guiding text as the search condition information; when the candidate content does not contain the video content or image content, filter at least one image corresponding to the guiding text from the auxiliary search images to obtain the search condition image, and use the search condition image and the guiding text as the search condition information.

[0024] In some embodiments, the generation unit may be specifically configured to, when the candidate content contains image content or video content, filter at least one image corresponding to the target search text from the candidate content and auxiliary search images to obtain an associated image, and use the associated image and the target search text as search condition information corresponding to the content search request; when the candidate content does not contain the image content or video content, filter at least one image corresponding to the target search text from the auxiliary search images to obtain an associated image, and use the associated image and the target search text as search condition information corresponding to the content search request.

[0025] In some embodiments, the generating unit may be specifically used to extract at least one content image and table information from the modal information of the candidate content; perform text extraction on the content image and the auxiliary search image to obtain current text information, and generate at least one table image based on the table information; and use the current text information, content image, table image and auxiliary search image as search condition information corresponding to the content search request.

[0026] In some embodiments, the content search device may further include a training unit, which may be used to acquire a training dataset, the training dataset including at least one sample pair, the sample pair including a text sample and an image sample corresponding to the text sample; construct sample triples corresponding to the sample pair, the sample triples including the sample pair and at least one nearest neighbor image sample corresponding to the text sample in the sample pair; and train a preset image generation model based on the sample triples to obtain the image generation model.

[0027] In some embodiments, the training unit may be specifically used to calculate the text similarity between the text sample and the image samples in the training dataset; based on the text similarity, to select at least one nearest neighbor image sample corresponding to the text sample from the image samples; and to add the nearest neighbor image sample to the sample pair corresponding to the text sample to obtain the sample triplet corresponding to the sample pair.

[0028] In some embodiments, the training unit may specifically be used to extract features from the text samples in the sample triplet to obtain sample text features, and to extract features from the neighboring image samples to obtain sample image features; to filter out target noise from a preset noise set, and to add noise to a preset image based on the target noise to obtain a noisy image; to denoise the noisy image using the preset image generation model based on the sample text features and sample image features, so as to obtain the target image corresponding to the text sample; and to converge the preset image generation model according to the target image and the image sample to obtain the image generation model.

[0029] In some embodiments, the retrieval unit may be specifically used to extract text information from the search prompt content to obtain the search text corresponding to the search prompt content; when the candidate content contains video content or image content, at least one image corresponding to the search text is selected from the preset image set and the candidate content to obtain an auxiliary search image; when the candidate content does not contain video content or image content, at least one image corresponding to the search text is selected from the preset image set to obtain an auxiliary search image.

[0030] In some embodiments, the generating unit may be specifically used to filter out at least one image to be edited from the candidate content and the auxiliary search image; determine the editing intent information of the image to be edited based on the target search text and the auxiliary search image; and perform image editing on the image to be edited based on the editing intent information to obtain the target search image corresponding to the content search request.

[0031] In some embodiments, the content search device may further include a search unit, which may be used to predict the current search content of at least one modality corresponding to the search prompt content using a graph network, wherein the graph network is constructed through a search prompt content sample and a search content sample of at least one modality corresponding to the search prompt content sample; and generate target search content corresponding to the content search request based on the current search content, wherein the target search content includes content of at least one modality.

[0032] Furthermore, this application also provides an electronic device, including a processor and a memory, wherein the memory stores an application program, and the processor is used to run the application program in the memory to execute the content search method provided in this application.

[0033] Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the content search methods provided in embodiments of this application.

[0034] Furthermore, embodiments of this application also provide a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps in the content search method provided in embodiments of this application.

[0035] In this embodiment, after obtaining a content search request, at least one candidate content is selected from a preset content set based on at least one search suggestion carried in the content search request. Modal information of at least one modality is extracted from the candidate content. Then, based on the modal information, target search text corresponding to the content search request is generated. At least one auxiliary search image is retrieved based on the search suggestion. A target search image corresponding to the content search request is generated based on the target search text and the auxiliary search image. Finally, the target search content corresponding to the target search request is determined based on the target search text and the target search image. This target search content includes content of at least one modality. Since this scheme can utilize the multimodal data of the candidate content selected from the search suggestion to generate the target search text, and can also retrieve auxiliary search images through the search suggestion to combine with the target search text to jointly generate the target search image, the consistency between the target search text and the target search image can be improved. Furthermore, since the auxiliary search image is introduced as additional knowledge during the generation of the target search image, the possibility of the target search image containing illusions is reduced. Therefore, the accuracy of the content search can be improved. Attached Figure Description

[0036] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram of a scenario for the content search method provided in an embodiment of this application;

[0038] Figure 2 This is a flowchart illustrating the content search method provided in an embodiment of this application;

[0039] Figure 3 This is a schematic diagram of the process of generating a target search image using a diffusion model, provided in an embodiment of this application.

[0040] Figure 4 This is a schematic diagram of the overall process of the content search method under the step category requirements provided in the embodiments of this application;

[0041] Figure 5This is another flowchart illustrating the content search method provided in the embodiments of this application;

[0042] Figure 6 This is a schematic diagram of the structure of the content search device provided in the embodiments of this application;

[0043] Figure 7 This is another structural schematic diagram of the content search device provided in the embodiments of this application;

[0044] Figure 8 This is another structural schematic diagram of the content search device provided in the embodiments of this application;

[0045] Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] This application provides a content search method and related equipment, which may include a content search device, an electronic device, a computer program product, and a computer-readable storage medium. The content search device may be integrated into an electronic device, which may be a server or a terminal, etc.

[0048] The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0049] For example, see Figure 1Taking the integration of a content search device into an electronic device as an example, after obtaining a content search request, the electronic device can filter at least one candidate content from a preset content set based on at least one search suggestion carried in the content search request, and extract modal information of at least one modality from the candidate content. Then, based on the modal information, it can generate the target search text corresponding to the content search request, and retrieve at least one auxiliary search image based on the search suggestion. Based on the target search text and the auxiliary search image, it can generate the target search image corresponding to the content search request. Then, based on the target search text and the target search image, it can determine the target search content corresponding to the target search request. The target search content includes content of at least one modality, thereby improving the accuracy of content search.

[0050] It is understood that, in the specific implementation of this application, data related to the preset content, candidate content, target search text, auxiliary search image, target search image and target search content of the object are involved. When the following embodiments of this application are applied to specific products or technologies, permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0051] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the preferred order of the embodiments.

[0052] This embodiment will be described from the perspective of a content search device, which can be integrated into an electronic device, such as a server or a terminal. The terminal can include tablet computers, laptops, personal computers (PCs), wearable devices, virtual reality devices, or other smart devices capable of content searching.

[0053] A content search method, comprising:

[0054] A content search request is obtained, which carries at least one search suggestion. Based on the search suggestion, at least one candidate content is selected from a preset content set, and modal information of at least one modality is extracted from the candidate content. Based on the modal information, target search text corresponding to the content search request is generated, and at least one auxiliary search image is retrieved based on the search suggestion. Based on the target search text and the auxiliary search image, target search image corresponding to the content search request is generated. Based on the target search text and the target search image, target search content corresponding to the target search request is determined, and the target search content includes content of at least one modality.

[0055] like Figure 2 As shown, Figure 2This is a schematic flowchart of an optional content search method provided in an embodiment of the present disclosure. The content search method can be executed by a server or a terminal alone, or it can be executed by a terminal in cooperation with a server. The specific flow of the content search method can be as follows:

[0056] 101. Obtain content search request.

[0057] The content search request includes at least one search suggestion. Search suggestions can be understood as information provided to aid in the content search process. The content type of search suggestions can be varied, including text, video, audio, or other formats. Taking text as an example, search suggestions might include the content title, keywords, content type, author, or other text that suggests searching for or generating specific / specified content. Search suggestions can also include longer text or charts, and so on.

[0058] There are several ways to obtain content search requests, as follows:

[0059] For example, it can receive content search requests sent by clients or terminals, which carry at least one search suggestion; or it can display a content search page, which includes at least one content input control. In response to an input operation on the content input control, it can obtain at least one search suggestion entered through the content input operation, and generate a content search request based on the search suggestion, which carries at least one search suggestion; or, when the content interaction platform receives a content search request sent by a client or terminal, it can receive a content search request sent by the server corresponding to the content interaction platform, which carries at least one search suggestion; or, when the number of search suggestions is large or the memory is large, it can also receive content search requests, which carry the storage address of at least one search suggestion, and obtain the search suggestion based on the storage address, and so on.

[0060] Wherein, "responding to" is used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed can be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0061] In this context, a content input control can be understood as a control that allows input or entry of content in at least one modality. Content input controls can be of various types, including text input boxes, audio / image / video input controls, or other controls that allow input / entry of content. Taking a text input box as an example, the target object can enter at least one search suggestion text in the text input box, using this search suggestion text as the search suggestion content, thereby triggering the generation of a content search request. This search request can carry the search suggestion text. It should be noted that a content search page can include one or more content input controls. For example, a content search page can accept text input, as well as uploaded video / audio / image content, and use these different modalities of content as search suggestion content.

[0062] 102. Based on the search suggestions, filter out at least one candidate content from the preset content set, and extract modal information of at least one modality from the candidate content.

[0063] In this context, a modality can be understood as an information carrier or a way of expressing information. Here, it can be understood as one type of information carrier within the candidate content. There can be various types of modalities, such as visual modalities, auditory modalities, linguistic modalities, or other types of information carriers. For example, taking a visual modality as an example, the modal information in the candidate content under this modality can include at least one of video information, image information, chart information, or other visually related information; taking an auditory modality as an example, the modal information in the candidate content under this modality can include at least one of audio information or other auditory-related information; taking a linguistic modality as an example, the modal information in the candidate content under this modality can include at least one of text information or other linguistic-related information, and so on.

[0064] The preset content set can be understood as a collection or database that pre-stores at least one type of content corresponding to a content search request. This preset content set can be a local content set, or a content set stored on a content platform or other third-party platform.

[0065] The process of selecting at least one candidate content from a preset content set based on search suggestions can be specifically described as follows:

[0066] For example, the search intent corresponding to the content search request can be identified from the search prompt information to obtain search intent information. Based on the search intent information, at least one preset content can be selected from the preset content set to obtain initial candidate content. At least one content segment can be segmented from the initial candidate content to obtain at least one candidate content.

[0067] Search intent information can be understood as information used to indicate the search intent of a content search request. Search intent can be understood as the goal or need that the content search request hopes to achieve. For example, taking the search suggestion "how to make scrambled eggs with tomatoes" as an example, the search intent of this search suggestion could include searching for the steps to make scrambled eggs with tomatoes or generating a recipe for scrambled eggs with tomatoes, and so on. There are several ways to identify the search intent corresponding to the content search request from the search suggestion information. For example, a semantic recognition model can be used to perform semantic recognition on the search suggestion content to obtain its semantic information, which can then be used as search intent information. Alternatively, when there are multiple search suggestions, a semantic recognition model can be used to perform semantic recognition on each suggestion to obtain its current semantic information, which can then be merged to obtain the search intent information. Another approach is to use a semantic recognition model to perform semantic recognition on the search suggestion content, obtain its semantic information, and then filter out the search intent corresponding to that semantic information from a preset set of search intents to obtain the search intent information corresponding to the content search request, and so on.

[0068] Semantic recognition models can be understood as models that perform semantic processing on search prompts. There are various types of semantic recognition models, such as knowledge graph models, CLIP (a multimodal semantic model), Word2Vec (a distributed semantic model), GloVe (a distributed semantic model), BERT (a deep learning semantic model), or other models capable of semantic processing. Taking an audio message with the semantic model BERT and the search prompt "how to make tomatoes" as an example, the methods for identifying the search intent corresponding to the content search request in the search prompt can include: converting the audio content into text content, using the BERT model to perform text segmentation to obtain at least one text word, encoding the text word to obtain at least one feature information of the text word, predicting the semantic information of the text content based on this feature information, and using this semantic information as the search intent information.

[0069] After identifying the search intent corresponding to the content search request from the search suggestions, at least one preset content can be selected from a preset content set based on the identified search intent information to obtain initial candidate content. The process of selecting at least one preset content from the preset content set may include: selecting at least one initial content corresponding to the search intent information from the preset content set; obtaining historical search information for the target object corresponding to the content search request; determining the content relevance between the initial content and the content search request based on the historical search information; and selecting at least one preset content from the initial content based on the content relevance to obtain initial candidate content.

[0070] There are several ways to filter out at least one initial content corresponding to the search intent information from the preset content set. For example, a coarse-sorting model can be used to filter out at least one initial content corresponding to the search intent information from the preset content set. Alternatively, the search intent information can be sent to the content server so that the content server can filter out at least one initial content corresponding to the search intent information from the preset content set and receive at least one initial content returned by the content server, and so on.

[0071] In this context, the coarse-ranking model can be understood as a model that quickly filters out a portion of high-potential, relevant content from a massive set of pre-defined content, providing input for subsequent fine-ranking. The coarse-ranking model can have various structures, such as a dual-tower model, a lightweight fully connected model, a multi-tower structure model, or other models that can quickly filter out high-potential content, etc. Taking the coarse-ranking model as an example of a dual-tower model, there are several ways to use the coarse-ranking model to select at least one initial content corresponding to the search intent information from a preset content set. For example, the search feature extraction network of the dual-tower model can be used to extract search features from the search intent information, and the content feature extraction network of the dual-tower model can be used to extract content features of preset content from the preset content set. The relevance score between the search features and content features is calculated to obtain the feature relevance parameter between the search intent information and the content in the preset content set. At least one preset content with a feature relevance parameter exceeding a preset threshold is then selected from the preset content set to obtain the initial content. Alternatively, the search feature extraction network of the dual-tower model can be used to extract search features from the search intent information, and the content feature extraction network of the dual-tower model can be used to extract content features of preset content from the preset content set. The relevance score between the search features and content features is calculated to obtain the feature relevance parameter between the search intent information and the content in the preset content set. Based on the feature relevance parameter, the preset content in the preset content set is sorted to obtain the sorting result. Based on the sorting result, at least one preset content within a preset sorting range is selected from the preset content set to obtain the initial content, and so on.

[0072] Historical search information can be understood as the search information of the target object that triggered the content search request between the current moment and the present moment. There are various types of historical search information, such as historical search behavior information, searched video metadata, and text information. There are several ways to obtain the historical search information of the target object corresponding to the content search request. For example, the object identifier of the target object that triggered the content search request can be extracted from the content search request, and the search information corresponding to the object identifier can be filtered from the historical search information set to obtain the historical search information of the target object. Alternatively, the object identifier of the target object that triggered the content search request can be extracted from the content search request, sent to the content server of the content platform, so that the content server can filter the historical search information corresponding to the object identifier from the historical search information set and receive the historical search information returned by the content server, and so on.

[0073] After filtering out at least one initial content corresponding to the search intent information and obtaining historical search information, the content relevance parameters between the initial content and the content search request can be determined based on the historical search information. There are several ways to determine the content relevance parameters between the initial content and the content search request based on historical search information. For example, a fine-ranking model can be used to extract features from historical search information to obtain historical search features. These historical search features can then be fused with the semantic features of the search suggestions to obtain target search features. The feature similarity or feature distance between the target search features and the content features of the initial content can then be calculated to obtain the content relevance parameters. Alternatively, a fine-ranking model can be used to extract features from both historical search information and search suggestions to obtain target search features. These fine-ranking features can then be used to extract features from the initial content to obtain current content features. The feature similarity or feature distance between the target search features and the current content features can then be calculated to obtain the content relevance parameters, and so on.

[0074] After determining the content relevance parameters between the initial content and the content search request, at least one preset content can be selected from the initial content based on these parameters to obtain initial candidate content. There are several ways to select at least one preset content from the initial content based on the content relevance parameters. For example, at least one preset content with a content relevance parameter exceeding a preset threshold can be selected to obtain initial candidate content. Alternatively, the initial content can be sorted based on the content relevance parameters, and the top N preset content can be selected based on the sorting results to obtain initial candidate content, where N is any integer greater than 2, and so on.

[0075] After selecting at least one preset content from a preset content set based on search intent information to obtain initial candidate content, at least one content segment can be segmented from the initial candidate content to obtain at least one candidate content. The method for segmenting at least one content segment from the initial candidate content may include: performing content slicing on the initial candidate content to obtain at least one candidate content segment; extracting features from the candidate content segments; performing semantic analysis on the candidate content segments based on the extracted content features to obtain semantic information of the candidate content segments; and selecting at least one candidate content segment from the candidate content segments based on the semantic information to obtain candidate content.

[0076] Taking video content as an example, there are several ways to slice the initial candidate content. For instance, the initial candidate content can be divided into multiple candidate video segments based on a preset time interval, with each candidate video segment having the same duration. Alternatively, the initial candidate content can be divided into frames to obtain at least one video frame. Content recognition can be performed on the video frames to obtain the content information of each video frame. Based on the content information, the initial candidate content can be sliced ​​to obtain at least one candidate video segment. Or, FFmpeg (a video processing tool) can be used to slice the initial candidate content to obtain at least one candidate content segment, and so on.

[0077] After slicing the initial candidate content, features can be extracted from the candidate content segments. Based on the extracted features, semantic analysis is then performed on the candidate content segments to obtain their semantic information. Taking video content as the initial candidate content as an example, the process of feature extraction and semantic analysis of candidate content segments can include: extracting features from the candidate content segments to obtain video features of the candidate video segments; performing speech recognition on the candidate video segments based on the video features to obtain the current text information corresponding to the candidate video segments; and performing semantic analysis on the current text information to obtain the semantic information of the candidate video segments.

[0078] After performing semantic analysis on the candidate content fragments, at least one candidate content fragment can be selected from them based on the semantic information. There are several ways to select at least one candidate content fragment based on semantic information. For example, one can calculate the correlation parameter between the semantic information of the candidate content fragment and the semantic information of the search suggestion content, rank the candidate content fragments based on the correlation parameter, and then select at least one candidate content fragment within a preset ranking range (e.g., TOP K, where K is any positive integer) based on the ranking results, thus obtaining candidate content. Alternatively, one can calculate the correlation parameter between the semantic information of the candidate content fragment and the semantic information of the search suggestion content, and then select at least one candidate content fragment whose correlation parameter exceeds a preset correlation parameter threshold, thus obtaining candidate content, and so on.

[0079] After filtering at least one candidate content from a preset content set based on search suggestions, at least one modal information can be extracted from the candidate content. There are various ways to extract at least one modal information from the candidate content. For example, taking video content as an example, text information, image information, video information, table information, audio information, or other modal information can be extracted from the candidate content, and at least one of these information can be used as modal information.

[0080] 103. Based on modal information, generate the target search text corresponding to the content search request, and retrieve at least one auxiliary search image according to the search prompts.

[0081] The target search text can be understood as the text information retrieved under the content-based search request, or as the search results or answers corresponding to the text type of the search suggestion. For example, if the search suggestion is "how to make scrambled eggs with tomatoes", the target search text could be the steps to make scrambled eggs with tomatoes, which could include step 1, step 2, ..., step n. Similarly, if the search suggestion is "how old is a certain soccer player XX?", the target search text could be "xx years old" or "soccer player XX is xx years old", and so on.

[0082] In this context, an auxiliary search image can be understood as an image that assists in generating the target search image corresponding to the target search text. Using auxiliary search images helps prevent the generated image from containing illusions during the target search image generation process.

[0083] The process of generating the target search text corresponding to the content search request based on modal information can be as follows:

[0084] For example, based on search intent information, content search requests can be categorized according to search needs to obtain the search need type of the content search request. When the search need type is a step list requirement, the target search text corresponding to the content search request can be generated based on modal information. When the search need type is a long text search requirement or a short text search requirement, the target search text corresponding to the content search request can be generated based on search suggestion information and modal information.

[0085] There are various types of search requests, including step-by-step list requests, short text search requests, and long text search requests. A step-by-step list request means the search query aims to find content that includes a list or at least one step within a list. For example, if the search suggestion is "how to make scrambled eggs with tomatoes," the search request type is a step-by-step list request. A long text search request means the search suggestion is a long text, or the desired search result is a long text. For example, if the search suggestion includes at least one document attachment and "analyze and summarize these documents," the search request type is a long text search request. A short text search request can be understood as any search request other than a long text search request. For example, if the search suggestion is "XX's requested age," the search request type is a short text search request.

[0086] Specifically, when the search request type is a step-by-step list request, the target search text corresponding to the content search request can be generated based on modal information. The methods for generating the target search text for the content search request can include: using a multimodal content generation model to extract features from the modal information, obtaining modal features for each modality, transforming the modal features to obtain transformed modal features, fusing the transformed modal features to obtain target text features, and generating the target search text corresponding to the content search request based on the target text features.

[0087] Specifically, when the search request type is a long text search request or a short text search request, the target search text corresponding to the content search request can be generated based on the search suggestion content and modal information. The process of generating the target search text corresponding to the content search request based on the search suggestion content and modal information can include: expanding the search suggestion content based on the search intent information to obtain expanded suggestion content; using a multimodal content generation model to extract features from the expanded suggestion content and modal information to obtain modal features and suggestion content features for each modality; converting the modal features and suggestion content features into feature information in the same feature space to obtain transformed features; fusing the transformed features to obtain target text features; and generating the target search text corresponding to the content search request based on the target text features.

[0088] There are several ways to expand search suggestions based on search intent information. For example, if the search suggestion is "the height of XX star player", it can be expanded to "how tall is XX star player". Expanding search suggestions is mainly to make them more complete, thereby improving the accuracy of the generated step-by-step list-style target text search results.

[0089] The method of retrieving at least one auxiliary search image based on the search prompt content can include: extracting text information from the search prompt content to obtain the search text corresponding to the search prompt content; when the candidate content contains video content or image content, selecting at least one image corresponding to the search text from the preset image set and the candidate content to obtain the auxiliary search image; when the candidate content does not contain video content or image content, selecting at least one image corresponding to the search text from the preset image set to obtain the auxiliary search image.

[0090] The methods for extracting text information from search suggestions can include: when the search suggestion content is text content, directly extracting the text information from the search suggestion content to obtain the search text; when the search suggestion content is content other than text, converting the search suggestion content into search suggestion text, extracting the text information from the search suggestion text to obtain the search text.

[0091] When candidate content includes video or image content, at least one image corresponding to the search text can be selected from a preset image set and candidate content to obtain an auxiliary search image set. The process of selecting at least one image corresponding to the search text from the preset image set and candidate content can include: extracting at least one search keyword from the search text; selecting at least one image corresponding to the search keyword from the preset image set to obtain a first candidate auxiliary search image; selecting at least one image corresponding to the search keyword from the candidate content to obtain a second candidate auxiliary search image; and merging the first and second candidate auxiliary search images to obtain at least one auxiliary search image. For example, taking the search text "how to make scrambled eggs with tomatoes" as an example, keywords such as "tomato," "scrambled eggs," and "make" can be extracted, and at least one image related to these keywords can be selected from the preset image set and candidate content to obtain auxiliary search images.

[0092] When the candidate content does not include video or image content, at least one image corresponding to the search text can be selected from a preset image set to obtain auxiliary search images. Methods for selecting at least one image corresponding to the search text from the preset image set may include: extracting at least one keyword from the search text, and selecting at least one image related to that keyword from the preset image set to obtain auxiliary search images.

[0093] It should be noted that a Retrieval-Augmented Generation (RAG) algorithm can be used to filter auxiliary search images from a preset content set and candidate content. This obtains visual content that matches the search requirements, thereby increasing the expressiveness of the final generated target search content.

[0094] 104. Generate the target search image corresponding to the content search request based on the target search text and the auxiliary search image.

[0095] In this context, the target search image can be understood as an image that matches the target search text. The number of target search images can be one or more. When the target search text includes multiple steps, the target search images can include the image corresponding to each step.

[0096] The method for generating the target search image corresponding to the internal search request based on the target search text and the auxiliary search image can be as follows:

[0097] For example, based on the type of search request and auxiliary search images, the search criteria information corresponding to the content search request can be determined. Based on the search criteria information, an image generation model can be used to generate at least one image corresponding to the target search text, thus obtaining the target search image.

[0098] The search condition information can be understood as the conditional information for generating the target search image. Search condition information can include text information and the corresponding image information. The process of determining the search condition information corresponding to the content search request based on the search demand type and the auxiliary search image can include: when the search demand type is a step list demand, identifying the guiding text for each step in the target search text, and determining the search condition information corresponding to the content search request based on the guiding text, the auxiliary search image, and candidate content; when the search demand type is a short text search demand, determining the search condition information corresponding to the content search request based on the target search text, the auxiliary search image, and candidate content; and when the search demand type is a long text search demand, determining the search condition information corresponding to the content search request based on candidate content and the auxiliary search image.

[0099] When the search request type is a step-by-step list request, the guiding text for each step is identified in the target search text. The guiding text can be understood as a sentence containing the core content or keywords of each step. There are several ways to identify the guiding text for each step in the target search text. For example, one can extract the step text corresponding to each step from the target search text, extract at least one keyword from the step text, and use the sentence containing that keyword as the guiding text. Alternatively, one can extract the step text corresponding to each step from the target search text, identify the sentence attribute information of each sentence in the step text, and based on the sentence attribute information, filter out the target sentence in the step text to obtain the guiding text, and so on.

[0100] After identifying the guiding text for each step in the target search text, the search criteria information corresponding to the content search request can be determined based on the guiding text, auxiliary search images, and candidate content. The process of determining the search criteria information corresponding to the content search request based on the guiding text, auxiliary search images, and candidate content can include: when the candidate content contains video or image content, at least one image corresponding to the guiding text is selected from the candidate content and auxiliary search images to obtain the search criteria image, and the search criteria image and guiding text are used as the search criteria information; when the candidate content does not contain video or image content, at least one image corresponding to the guiding text is selected from the auxiliary search images to obtain the search criteria image, and the search criteria image and guiding text are used as the search criteria information.

[0101] When candidate content includes video or image content, it means that images can be selected from the candidate content. Therefore, the search condition image can be selected from both the candidate content and the auxiliary search images. There are several ways to select at least one image corresponding to the guiding text from the candidate content and auxiliary search images. For example, guiding keywords can be extracted from the guiding text, and at least one image corresponding to the guiding keywords can be selected from the candidate content and auxiliary search images to obtain the search condition image. Alternatively, feature extraction can be performed on the guiding text to obtain guiding text features, and image feature extraction can be performed on the candidate content and auxiliary search images to obtain current image features. The feature similarity or feature distance between the guiding text features and the current image features can be calculated to obtain the image-text relevance. This image-text relevance indicates the degree of association between the guiding text and the images in the candidate content and the auxiliary search images. Based on the image relevance, at least one image can be selected from the candidate content and auxiliary search images to obtain the search condition image, and so on.

[0102] When the candidate content does not contain video content or images, it means that the search condition image cannot be filtered from the candidate content. In this case, the condition image can only be filtered from the auxiliary search images. The method for filtering the search condition image from the auxiliary search images is similar to the method for filtering the search condition image from the candidate content and auxiliary search images, as described above, and will not be repeated here.

[0103] Specifically, when the search request type is a short text search request, the search criteria information corresponding to the content search request can be determined based on the target search text, auxiliary search images, and candidate content. The process of determining the search criteria information corresponding to the content search request based on the target search text, auxiliary search images, and candidate content can include: when the candidate content contains image content or video content, at least one image corresponding to the target search text is selected from the candidate content and auxiliary search images to obtain a related image, and the related image and the target search text are used as the search criteria information corresponding to the content search request; when the candidate content does not contain image content or video content, at least one image corresponding to the target search text is selected from the auxiliary search images to obtain a related image, and the related image and the target search text are used as the search criteria information corresponding to the content search request.

[0104] When candidate content includes image or video content, it means that images can be extracted from the candidate content. Therefore, images associated with the target search text can be selected from both candidate content and auxiliary search images. There are several ways to select at least one image corresponding to the target search text from candidate content and auxiliary search images. For example, text features can be extracted from the target search text to obtain search text features. Current image features can be selected from candidate content and auxiliary search images. The similarity and feature distance between the search text features and current image features can be calculated to obtain feature correlation. Based on feature correlation, at least one image can be selected from candidate content and auxiliary search images to obtain an associated image. Alternatively, at least one keyword can be extracted from the target search text. Based on the keyword, at least one image can be selected from candidate content and auxiliary search images to obtain an associated image, and so on.

[0105] After filtering out the related images, the related images and the target search text can be used as search criteria for the content search request.

[0106] When the candidate content does not contain image or video content, it means that no image can be extracted from the candidate content. Therefore, related images can only be selected from the auxiliary search images. The method for selecting related images from the auxiliary search images is similar to the method for selecting related images from the candidate content and auxiliary search images, as described above, and will not be repeated here.

[0107] Specifically, when the search request type is a long text search request, the search criteria information corresponding to the content search request can be determined based on candidate content and auxiliary search images. The process of determining the search criteria information corresponding to the content search request based on candidate content and auxiliary search images may include: extracting at least one content image and table information from the modal information of the candidate content; extracting text from the content image and auxiliary search images to obtain the current text information; generating at least one table image based on the table information; and using the current text information, content image, table image, and auxiliary search image as the search criteria information corresponding to the content search request.

[0108] Table information can be understood as table-related information within the candidate content. Table information can be of various types, including content inserted within the table (such as text, images, or videos), table format, location, title, and type (such as a regular table, bar chart, pie chart, or other table types), etc. The process of extracting at least one content image and at least one table information from the modal information of the candidate content can include: querying the modal information for both image and table modalities; when image modal information exists, extracting at least one content image from it; and when table modal information exists, extracting the table information from it.

[0109] Once the table information is extracted, at least one table image can be generated based on it. There are several ways to generate at least one table image. For example, the position of the table in the content image or candidate content can be identified from the table information to obtain the table position. Then, the region corresponding to the table position can be extracted from the content image or candidate content to obtain at least one table image. Alternatively, a table template image corresponding to the table information can be selected from a preset set of table template images, and the table information can be inserted into the table template image to obtain at least one table information. Or, at least one image can be cropped from the content image based on the table information to obtain the table image, and so on.

[0110] There are several ways to extract text from content images and auxiliary search images. For example, an OCR model can be used to identify at least one text character in the content image and auxiliary search image to obtain the current text information. Alternatively, a text feature extraction model can be used to extract text features from the content image and auxiliary search image to obtain the current text features. Based on the current text features, the text information in the content image and auxiliary search image can be determined to obtain the current text information, and so on.

[0111] After extracting the current text information and generating at least one table image, the current text information, content image, table image, and auxiliary search image can be used as search criteria information corresponding to the content search request.

[0112] After determining the search criteria information corresponding to the content search request based on the search demand type and auxiliary search images, at least one image corresponding to the target search text can be generated using an image generation model based on the search criteria information, thereby obtaining the target search image. The process of generating at least one image corresponding to the target search text using an image generation model may include: extracting the current condition text and current condition image from the search criteria information; extracting features from the current condition text using an image generation model to obtain search text features; extracting features from the current condition image using an image generation model to obtain search image features; sampling current noise from a preset noise set; adding noise to the preset image based on the current noise to obtain a current noise image; and denoising the current noise image using an image generation model based on the search image features and search text features to obtain the target search image corresponding to the target search text.

[0113] The process of extracting the current condition text and current condition image from the search condition information can include: when the search condition information includes guiding text and search condition image, the guiding text can be used as the current condition text and the search condition image can be used as the current condition image; when the search condition information includes related images and target search text, the target search text can be used as the current condition text and the related images can be used as the current condition image; when the search condition information includes current text information, content image, table image and auxiliary search image, the current text information can be used as the current condition text and the content image, table image and auxiliary search image can be used as the current condition image.

[0114] In this context, the noise in the preset noise set can be understood as information that interferes with the image content. There are various types of noise, such as Gaussian noise, salt-and-pepper noise, Poisson noise, fixed-pattern noise, or other types. There are several ways to sample the current noise from the preset noise set. For example, noise can be randomly sampled from the preset noise set to obtain the current noise, or Gaussian sampling can be performed on the noise in the preset noise set to obtain the current noise, and so on.

[0115] After sampling the current noise from a preset noise set, the preset image can be denoised based on the current noise to obtain a current noisy image. There are several ways to denoise the preset image based on the current noise. For example, the noise value at at least one pixel position of the current noise can be added to the corresponding pixel value in the preset image to obtain the current noisy image. Alternatively, the noise value at at least one pixel position can be extracted from the current noise, the pixel value at that position can be identified in the preset image, and the pixel value at that position can be replaced with the noise value to obtain the current noisy image, and so on.

[0116] Then, based on the search image features and search text features, an image generation model is used to denoise the current noisy image to obtain the target search image corresponding to the target search text. The process of denoising the current noisy image using an image generation model may include: denoising the current noisy image based on the search image features and search text features to obtain a denoised image; returning to the step of sampling the current noise from a preset noise set; and repeating this process until a preset number of denoising iterations is reached, thereby obtaining the target search image corresponding to the target search text.

[0117] Taking the image generation model as a diffusion model and video content as the candidate content as an example, the process of generating the target search image corresponding to the target search text using the diffusion model can be as follows: Figure 3 As shown, based on the input text information (current condition text) and the image information (current condition image) selected from the video content or auxiliary search images, a text encoding network is used to encode the current condition text to obtain the search text features of the current condition text. An image encoding network is used to encode the current condition image to obtain the search image features of the current condition image. The search text features and search image features are used as input to the diffusion model. Starting from random noise, the diffusion model predicts the noise added at each step based on the number of input steps, search text features, and search image features. By using the predicted noise, the image with added noise can be gradually converted into a clear image, thus obtaining the target search image corresponding to the target search text. This target search image can be understood as generating an image that conforms to the relationship between the text description (target search image) and the real image (at the auxiliary search image), thereby improving the consistency between the target search text and the target search image, and thus improving the accuracy of content search.

[0118] Optionally, in some embodiments, the image generation model can be set by maintenance personnel or trained by the content search device itself. That is, before the step "obtaining the target search image by using at least one image corresponding to the image generation model based on search condition information", the content search method may further include:

[0119] Obtain a training dataset that includes at least one sample pair, each consisting of a text sample and its corresponding image sample. Construct sample triplets for each sample pair, each consisting of the sample pair and at least one nearest neighbor image sample corresponding to the text sample in the sample pair. Based on these sample triplets, train a pre-defined image generation model to obtain the image generation model, as follows:

[0120] (1) Obtain the training dataset

[0121] The training dataset includes at least one sample pair, which includes a text sample and an image sample of the text sample pair.

[0122] There are several ways to obtain the training dataset, including the following:

[0123] For example, a training dataset uploaded by a client or terminal can be obtained, which includes at least one sample pair, which includes a text sample and an image sample corresponding to the text sample. Alternatively, at least one text sample can be obtained, and the image sample corresponding to the text sample can be selected from a preset image set to obtain at least one sample pair, and so on.

[0124] (2) Construct sample triplets corresponding to sample pairs

[0125] The sample triplet consists of a sample pair and at least one nearest-neighbor image sample corresponding to the text sample in the sample pair. The sample triplet can include <text sample, image sample, nearest-neighbor image sample>.

[0126] The process of constructing the sample triplet corresponding to the sample pair can be described as follows:

[0127] For example, the text similarity between a text sample and an image sample in the training dataset can be calculated. Based on the text sample similarity, at least one nearest neighbor image sample corresponding to the text sample can be selected from the image samples. The nearest neighbor image sample can be added to the sample pair corresponding to the text sample to obtain the sample triplet corresponding to the sample pair.

[0128] Text similarity can be understood as the similarity between text samples and image samples, and this similarity is reflected in the text dimension. There are several ways to calculate the text similarity between a text sample and an image sample in the training dataset. For example, features can be extracted from the text sample to obtain sample text features, and features can be extracted from the image samples in the training dataset to obtain candidate sample image features. The feature similarity or feature distance between the sample text features and the candidate sample image features can then be calculated to obtain the text similarity. Alternatively, features can be extracted from the text sample to obtain sample text features, and features can be extracted from the image samples in the training dataset to obtain candidate sample image features. The candidate sample image features can then be converted into text-dimensional features to obtain the converted features, and the feature similarity or feature distance between the sample text features and the converted features can then be calculated to obtain the text similarity, and so on.

[0129] After calculating the text similarity between the text sample and the image samples in the training dataset, at least one nearest neighbor image sample corresponding to the text sample can be selected from the image samples based on the text similarity. There are several ways to select at least one nearest neighbor image sample corresponding to the text sample from the image samples. For example, at least one image sample with a text similarity exceeding a preset similarity threshold can be selected from the image samples to obtain at least one nearest neighbor image sample. Alternatively, the image samples can be sorted based on the text similarity to obtain a sorting result. Based on the sorting result, at least one image sample within a preset sorting range (e.g., TOP N, where N is any positive integer) can be selected from the image samples to obtain at least one nearest neighbor image sample, and so on.

[0130] After filtering out at least one nearest neighbor image sample corresponding to the text sample from the image samples, the nearest neighbor image sample can be added to the sample pair corresponding to the text sample to obtain the sample triplet corresponding to the sample pair.

[0131] The sample triples mainly consist of text samples, image samples, and nearest neighbor image samples. This data structure provides rich contextual information for subsequent model training, ensuring that the generated images better reflect the relationship between the text content (target search text) and related images (search condition images / auxiliary search images / candidate content), thereby improving the accuracy of content search.

[0132] After constructing sample triplets corresponding to sample pairs, the preset image generation model can be trained using these sample triplets to obtain the image generation model. The process of training the preset image generation model based on the sample triplets can include: extracting features from the text samples in the sample triplets to obtain sample text features; extracting features from the nearest neighbor image samples to obtain sample image features; selecting target noise from a preset noise set; adding noise to the preset image based on the target noise to obtain a noisy image; using the preset image generation model to denoise the noisy image based on the sample text features and sample image features to obtain the target image corresponding to the text sample; and converging the preset image generation model based on the target image and the image sample to obtain the image generation model.

[0133] The method of selecting target noise from the preset noise set is similar to the method of selecting current noise from the preset noise set, as described above, and will not be repeated here.

[0134] After selecting the target noise from the preset noise set, the preset image can be denoised based on the target noise to obtain the target image corresponding to the text sample. The method of denoising the preset image based on the target noise is similar to the method of denoising the preset image based on the current noise, as described above, and will not be repeated here.

[0135] After adding noise to the preset image based on the target noise, the noisy image can be denoised using a preset image generation model based on the sample text features and sample image features, thereby obtaining the target image corresponding to the text sample. The process of denoising the noisy image using the preset image generation model can include: denoising the noisy image based on the sample text features and sample image features; using the denoised image as the preset image; returning to the preset noise set to filter out the target noise; and repeating this process until a preset number of denoising steps is reached, thereby obtaining the target image corresponding to the text sample.

[0136] The process of denoising a noisy image using a preset image generation model based on sample text features and sample image features may include: predicting the noise added to the noisy image using a preset image generation model based on sample text features and sample image features to obtain predicted noise data, and denoising the noisy image based on the predicted noise data to obtain the current denoised image.

[0137] Based on the features of the sample text and sample images, a pre-defined image generation model is used to denoise the noisy image to obtain the target image corresponding to the text sample. Then, the pre-defined image generation model can be converged based on the target image and image samples to obtain the final image generation model. There are several ways to converge the pre-defined image generation model based on the target image and image samples. For example, based on the target image and image samples, the image loss of the sample triples can be determined. Based on the image loss, the gradient descent algorithm can be used to update the network parameters of the pre-defined image generation model to obtain the updated image generation model. This updated image generation model is then used as the pre-defined image generation model, and the process of obtaining the training dataset is repeated until the pre-defined image generation model converges. Alternatively, based on the target image and image samples, the image loss of the sample triples can be determined. Based on the image loss, other model parameter update algorithms can be used to update the network parameters of the pre-defined image generation model to obtain the updated image generation model. This updated image generation model is then used as the pre-defined image generation model, and the process of obtaining the training dataset is repeated until the pre-defined image generation model converges. And so on.

[0138] Image loss can be understood as loss information indicating the difference between the generated target image and the image samples. There are various ways to determine the image loss based on the target image and the image samples, specifically the sample triples. For example, features can be extracted from the target image to obtain target image features, and the feature distance or feature similarity between the target image features and the sample images can be calculated to obtain the image loss. Alternatively, the pixel difference between the target image and the image samples at each pixel location can be calculated and fused to obtain the image loss. Another approach is to calculate the squared difference of the pixel values ​​at each pixel location between the target image and the image samples, and so on.

[0139] Taking the pre-defined image generation model as a diffusion model as an example, the overall structure of the diffusion model includes multiple layers of noise prediction and image generation networks. During training, the diffusion model can start with random noise and gradually predict the noise added at each step based on the number of input steps, text information (text samples), and searched nearest neighbor image samples. The diffusion model learns how to gradually transform noise into a clear image through backpropagation, while ensuring that the generated image maintains consistency with the input text samples and nearest neighbor images to enhance the semantic relevance of the generated image.

[0140] Optionally, in some embodiments, in addition to generating the target search image using an image generation model, existing images can also be edited to obtain the target search image. The process of editing existing images may include: selecting at least one image to be edited from candidate content and auxiliary search images; determining the editing intent information of the image to be edited based on the target search text and auxiliary search images; and performing image editing based on the editing intent information to obtain the target search image corresponding to the content search request.

[0141] The method of selecting at least one image to be edited from candidate content and auxiliary search images may include: when the candidate content contains image content or video content, selecting at least one image corresponding to the target search text from the candidate content and auxiliary search images to obtain the image to be edited; when the candidate content does not contain image content or video content, selecting at least one image corresponding to the target search text from the auxiliary search images to obtain the image to be edited.

[0142] The method of selecting the image to be edited from the candidate content and auxiliary search images is similar to the method of selecting the image under the current conditions from the candidate content and auxiliary search images, as described above, and will not be repeated here.

[0143] After selecting at least one image to be edited from candidate content and auxiliary search images, the editing intent information of the image to be edited can be determined based on the target search text and the auxiliary search images. Editing intent information can be understood as information indicating instructions for editing the image. For example, it could include "replace the red flowers in the image with blue flowers," "remove objects other than tomatoes and eggs from the tomato and egg stir-fry image," or "change the image style to XX style," or other instructions, etc. The methods for determining the editing intent information of the image to be edited based on the target search text and auxiliary search images can include: extracting semantic features from the target search text to obtain the current search semantic features; extracting semantic features from the image to be edited to obtain the current semantic features; comparing the current search semantic features and the current semantic features to obtain the initial editing intent information; when the image to be edited is an image in the auxiliary search images, using the initial editing intent information as the editing intent information; when the image to be edited is not an image outside the auxiliary search images, extracting image features from the auxiliary search images to obtain the current auxiliary image features; and adjusting the initial editing intent information based on the current auxiliary image features to obtain the editing intent information of the image to be edited.

[0144] After determining the editing intent information of the image to be edited based on the target search text and auxiliary search images, image editing can be performed on the image to be edited based on the editing intent information to obtain the target search image corresponding to the content search request. The image editing process may include: using an image editing model to identify the task type of the editing task and the target editing object corresponding to the editing task from the editing intent information; detecting or segmenting the region to be edited in the image to be edited based on the task type and target editing object; generating the editing content corresponding to the editing task according to the editing intent information; and fusing the edited content with the original content in the region to be edited to obtain the target search image corresponding to the content search request.

[0145] For example, taking the editing task of "replacing red flowers in an image with blue flowers", the image editing model can generate the texture of blue flowers, and then replace the texture of the area where the red flowers are located in the image with the texture of the generated blue flowers, thus obtaining the edited image.

[0146] 105. Based on the target search text and target search image, determine the target search content corresponding to the target search request.

[0147] The target search content can include content of at least one modality. For example, if the target search content includes images and text, then the target search content can be understood as content with both images and text.

[0148] There are several ways to determine the target search content corresponding to the target search request based on the target search text and the target search image, as follows:

[0149] For example, the target search text and the target search image can be fused to obtain the target search content corresponding to the content search request. Alternatively, based on the target search text and the target search image, a multimodal large language model can be used to generate the target search content corresponding to the content search request, which includes content in at least one modality, and so on.

[0150] There are several ways to merge target search text and target search image. For example, you can filter out the content display frame corresponding to the content search request from a set of preset content display frames, and add the target search text and target search image to the content display frame to obtain the target search content. Alternatively, when the target search text includes text for multiple steps, you can filter out the image corresponding to each step from the target search images, and combine the target search text and target search image in the order of the steps to obtain the target search content.

[0151] The method of generating target search content corresponding to a content search request using a multimodal large language model based on target search text and target search image may include: using a text encoding network in the multimodal large language model to encode the target search text to obtain target text features; using an image encoding network in the multimodal large language model to encode the target search image to obtain target image features; performing feature transformation on the target text features and target image features to obtain transformed text features corresponding to the target text features and transformed image features corresponding to the target image features; performing semantic analysis on the transformed text features and transformed image features to obtain text semantic information corresponding to the transformed text features and image semantic information corresponding to the transformed image features; fusing the text semantic information and image semantic information to obtain multimodal semantic information; generating modal content of at least one modality based on the multimodal semantic information; and fusing the modal content to obtain the target search content, which includes content of at least one modality.

[0152] Multimodal large language models can be understood as language models that generate multimodal content from data of multiple modalities. There are various types of multimodal large language models, such as NExT-GPT (a type of multimodal large language model), Qwen2.5-VL (a type of multimodal large language model), or other models capable of generating multimodal content, etc. Taking NExT-GPT as an example of a multimodal large language model, the process of generating target search content based on target search text and target search image can include: using a high-performance unified encoding (e.g., ImageBind (an encoder)) to encode the input (target search text and target search image) under different modalities, and mapping the encoded input to a representation (i.e., features) that the large language model can understand through a projection layer; using an LLM network as the core, receiving the multimodal encoded features, performing semantic understanding and reasoning, and outputting text response or other modal signal tags; using the output projection layer of Transformer to convert the signal tags into a form that the multimodal decoder can understand, and generating the final multimodal content through a conditional diffusion model (e.g., StableDiffusion for images and Zeroscope for videos), thereby obtaining target search content containing modal content of at least one modality.

[0153] After determining the target search content corresponding to the target search request based on the target search text and target search image, the target search content can be displayed. There are several ways to display the target search content. For example, it can display a content display page corresponding to the content search request, which can show the target search content; or, it can extract the content display address from the content search request and send the target search content to the display terminal based on the content display address; or, it can extract the target object's terminal information from the content search request and send the target search content to the target object's terminal based on the terminal information; or, it can send the target search content to the interactive server of the content platform corresponding to the content search request, so that the interactive server can send the target search content to at least one interactive terminal, and so on.

[0154] Optionally, in some embodiments, after obtaining the content search request, a knowledge graph can be used to infer the multimodal information corresponding to the content search request and generate the target search content. The process of using a knowledge graph for content search may include: using a graph network to predict the current search content of at least one modality corresponding to the search prompt information, and generating the target search content corresponding to the content search request based on the current search content, wherein the target search content includes content of at least one modality.

[0155] The graph network is constructed using search suggestion content samples and modal content samples of at least one modality corresponding to the search suggestion content samples. Taking the search suggestion content samples as the user question and the content search request as the video search request, a knowledge graph (i.e., a graph network) can be constructed that contains video content, related knowledge (information of at least one modality related to the video content), and the user question (search suggestion content). The process of constructing a graph network may include: acquiring a set of video data samples, which includes at least one video and related data of the video (e.g., text, images, or audio related to the video); acquiring search data samples, which includes historical search suggestion samples (user questions) of at least one sample object and historical search content samples (answers to user questions) corresponding to the historical search suggestion samples; identifying at least one entity in the video data sample set and the search data sample set, and identifying the relationships between entities (e.g., video A contains person B, or the answer to question A is video B, etc.); extracting the attribute information of the entities from the video data samples and the search data samples (e.g., video duration, resolution, question category, etc.); and using entities as nodes and the relationships between entities as edges between nodes, thereby constructing a graph network (knowledge graph).

[0156] After constructing the graph network, it can be used to predict the current search content of at least one modality corresponding to the search prompt information. The process of using a graph network to predict the current search content of at least one modality corresponding to the search prompt information may include: extracting features from the search prompt content to obtain the current content features of the search prompt content; and using the graph network to filter out the current search content of at least one modality from the video data samples based on the current content features.

[0157] After using a graph network to predict the current search content corresponding to at least one modality of the search suggestion, the target search content corresponding to the content search request can be generated based on the current search content. The target search content may include content of at least one modality. The method of generating the target search content based on the current search content is similar to the method of determining the target search content based on the target search text and the target search image, as described above, and will not be repeated here.

[0158] Taking a video search request as an example, a search suggestion as a query text, and a search request type as a step-by-step category request, the overall content search process of this solution can be described as follows: Figure 4As shown, the system receives the query text input by the user and understands the user's search intent using natural language processing (NLP) technology. Then, a coarse-ranking model is used to initially filter massive amounts of video data, quickly identifying a subset of potentially relevant videos. Based on the parsed query text, and combining the user's historical behavior data, video metadata, and text information, personalized video retrieval is further performed. The top ten videos with the highest relevance to the user's query are selected from the coarse-ranked videos, forming preliminary video retrieval results. These retrieved videos undergo in-depth processing, including slicing, feature extraction, automatic speech recognition (ASR), and semantic analysis. Through slicing technology, the video is divided into multiple segments and associated with the most relevant segments of the user's query, resulting in candidate videos. Based on the user's query intent, the search requirements corresponding to the query text are categorized into step list requirements, short text answer requirements, and long text answer requirements. Depending on the type of search requirement, multimodal information (e.g., text, images, videos, tables, or other information) is extracted from the candidate content. Additionally, RAG technology can be used to retrieve images corresponding to various search requirements (auxiliary search images). After extracting multimodal information, a solution can be generated using a multimodal large language model based on the extracted multimodal information and retrieved auxiliary search images. This solution includes text for multiple steps, thus obtaining the target search text. Based on the generated target search text and auxiliary search images, image retrieval enhancement techniques are used to generate images corresponding to each step, thus obtaining the target search image corresponding to the target search text. Finally, the multimodal large language model is used to fuse the target search text and target search image to obtain the multimodal search result, i.e., the target search content.

[0159] Taking a content search scenario as an example, this solution extracts and enhances video information, combined with multimodal retrieval technology, to provide users with more comprehensive and accurate answers. Through this multimodal retrieval generation method, users not only obtain direct answers but also gain a deeper understanding of the question's background and details through the combination of relevant video clips, images, and text information. Furthermore, this technical solution can adapt to the needs and preferences of different users, providing a personalized search experience, further enhancing user satisfaction and interactivity, and offering users a more intuitive and vivid way to acquire knowledge. In addition, this solution introduces auxiliary search images as additional knowledge transfer into the image generation model during the generation of the target search image. Real images can be used as relevant information input to the text-to-image model. Combined with the answer text, images that match the search results are generated, making the search results more realistic and reducing the illusion problem in image generation.

[0160] It's also worth noting that this content search method can be applied not only to video search and Q&A scenarios, but also to chatbots, AI search, intelligent customer service, and many other applications. The core of this solution lies in integrating multiple modalities of information (such as text, images, and audio) to provide users with richer and more intuitive multimodal responses. This flexibility allows the technology to adapt to different application needs, enhance user experience, and meet diverse information retrieval requirements. Furthermore, the final generated target search content can not only display richly illustrated content, but also content in at least one other modality.

[0161] As can be seen from the above, after obtaining a content search request, this embodiment of the application filters at least one candidate content from a preset content set based on at least one search suggestion carried in the content search request, extracts modal information of at least one modality from the candidate content, then generates target search text corresponding to the content search request based on the modal information, retrieves at least one auxiliary search image based on the search suggestion, generates a target search image corresponding to the content search request based on the target search text and the auxiliary search image, and then determines the target search content corresponding to the target search request based on the target search text and the target search image. The target search content includes content of at least one modality. Since this scheme can use the multimodal data of the candidate content filtered by the search suggestion to generate target search text, and then retrieve auxiliary search images through the search suggestion to combine with the target search text to jointly generate the target search image, the consistency between the target search text and the target search image can be improved. Moreover, since the auxiliary search image is introduced as additional knowledge in the process of generating the target search image, the possibility of the target search image containing illusions is reduced, thus improving the accuracy of content search.

[0162] Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

[0163] In this embodiment, the content search device will be specifically integrated into an electronic device, the electronic device will act as a server, the content search request will be a video search request, and the candidate content will be candidate video content, as an example for explanation.

[0164] like Figure 5 As shown, a content search method has the following specific process:

[0165] 201. The server receives a video search request.

[0166] For example, a server can receive a video search request sent by a client or terminal, the video search request carrying at least one search suggestion; or, it can display a content search page including at least one content input control, and in response to an input operation on the content input control, obtain at least one search suggestion input through the content input operation, and generate a video search request based on the search suggestion, the video search request carrying at least one search suggestion; or, when a content interaction platform receives a video search request sent by a client or terminal, it can receive a video search request sent by the server corresponding to the content interaction platform, the video search request carrying at least one search suggestion; or, when the number of search suggestion items is large or the memory is large, it can also receive a video search request carrying the storage address of at least one search suggestion, and obtain the search suggestion based on the storage address, and so on.

[0167] 202. Based on the search suggestions, the server selects at least one candidate video from a preset content set.

[0168] For example, a server can use a semantic recognition model to perform semantic recognition on search suggestions to obtain semantic information of the search suggestions, and use this semantic information as search intent information. Alternatively, when there are multiple search suggestions, a semantic recognition model can be used to perform semantic recognition on the search suggestions to obtain the current semantic information of each search suggestion, and then the current semantic information can be fused to obtain search intent information. Or, a semantic recognition model can be used to perform semantic recognition on the search suggestions to obtain semantic information of the search suggestions, and then the search intent corresponding to the semantic information can be selected from a preset set of search intents to obtain the search intent information corresponding to the video search request, and so on.

[0169] The server can use a coarse-sorting model to filter out at least one initial video corresponding to the search intent information from a preset content set. Alternatively, the search intent information can be sent to the content server so that the content server can filter out at least one initial video corresponding to the search intent information from the preset video set, and receive at least one initial video returned by the content server, and so on.

[0170] The server can extract the object identifier of the target object that triggered the video search request from the video search request, filter the search information corresponding to the object identifier from the historical search information set, and thus obtain the historical search information of the target object. Alternatively, it can extract the object identifier of the target object that triggered the video search request from the video search request, send the object identifier to the content server of the content platform, so that the content server can filter the historical search information corresponding to the object identifier from the historical search information set, and receive the historical search information returned by the content server, and so on.

[0171] The server can use a fine-ranking model to extract features from historical search information to obtain historical search features. These historical search features are then fused with the semantic features of the search suggestions to obtain the target search features. The similarity or distance between the target search features and the content features of the initial content is calculated to obtain content relevance parameters. Alternatively, the server can use a fine-ranking model to extract features from both historical search information and search suggestions to obtain the target search features. These fine-ranking features are then used to extract features from the initial content to obtain the current content features. The similarity or distance between the target search features and the current content features is then calculated to obtain content relevance parameters, and so on.

[0172] The server can filter out at least one preset content from the initial video whose content relevance parameter exceeds a preset threshold, thereby obtaining the initial candidate video. Alternatively, it can sort the initial content according to the content relevance parameter and, based on the sorting result, filter out the TOP N preset content from the initial content to obtain the initial candidate video, where N is any integer greater than 2, and so on.

[0173] The server can divide the initial candidate video into multiple candidate video segments based on a preset time interval, with each candidate video segment having the same time interval. Alternatively, the server can divide the initial candidate video into frames to obtain at least one video frame, perform content recognition on the video frame to obtain the content information of each video frame, and divide the initial candidate video into segments based on the content information to obtain at least one candidate video segment. Alternatively, the server can use FFmpeg to divide the initial candidate video into segments to obtain at least one candidate video segment, and so on.

[0174] The server extracts features from candidate video segments to obtain video features. Based on these video features, it performs speech recognition on the candidate video segments to obtain the current text information corresponding to the candidate video segments. Finally, it performs semantic analysis on the current text information to obtain the semantic information of the candidate video segments.

[0175] The server can calculate the correlation parameters between the semantic information of candidate video segments and the semantic information of search suggestions, sort the candidate video segments based on the correlation parameters, and select at least one candidate video segment from the candidate video segments that falls within a preset sorting range (e.g., TOP K, where K is any positive integer) based on the sorting results, thereby obtaining candidate videos. Alternatively, it can calculate the correlation parameters between the semantic information of candidate video segments and the semantic information of search suggestions, and select at least one candidate video segment from the candidate video segments whose correlation parameters exceed a preset correlation parameter threshold, thereby obtaining candidate videos, and so on.

[0176] 203. The server extracts modal information of at least one modality from the candidate video.

[0177] For example, the server can extract text information, image information, video information, table information, audio information, or other modal information from candidate videos, and use at least one of these information as modal information.

[0178] 204. The server generates the target search text corresponding to the video search request based on modal information. For example, based on search intent information, the server classifies video search requests according to search needs to obtain the search need type of the video search request. When the search need type is a step list requirement, a multimodal content generation model is used to extract features from the modal information to obtain modal features for each modality. The modal features are then transformed to obtain transformed modal features. The transformed modal features are then fused to obtain target text features. Based on the target text features, the target search text corresponding to the video search request is generated. When the search need type is a long text search requirement or a short text search requirement, the search suggestion content is expanded based on the search intent information to obtain expanded suggestion content. A multimodal content generation model is used to extract features from the expanded suggestion content and modal information to obtain modal features and suggestion content features for each modality. The modal features and suggestion content features are then converted into feature information in the same feature space to obtain transformed features. The transformed features are then fused to obtain target text features. Based on the target text features, the target search text corresponding to the video search request is generated.

[0179] 205. The server retrieves at least one auxiliary search image based on the search suggestions.

[0180] For example, when the search suggestion content is text, the server can directly extract the text information from the search suggestion content to obtain the search text. When the search suggestion content is content other than text, the server can convert the search suggestion content into search suggestion text, extract the text information from the search suggestion text, and obtain the search text.

[0181] When a candidate video contains video or image content, the server extracts at least one search keyword from the search text, selects at least one image corresponding to the search keyword from a preset image set to obtain a first candidate auxiliary search image, selects at least one image corresponding to the search keyword from the candidate video to obtain a second candidate auxiliary search image, and merges the first and second candidate auxiliary search images to obtain at least one auxiliary search image. When a candidate video does not contain video or image content, the server extracts at least one keyword from the search text, selects at least one image related to that keyword from a preset image set to obtain an auxiliary search image.

[0182] 206. The server generates the target search image corresponding to the video search request based on the target search text and auxiliary search images.

[0183] For example, when the search request type is a list of steps, the server can extract the step text corresponding to each step from the target search text, extract at least one keyword from the step text, and use the sentence containing the keyword as the guiding text. Alternatively, it can extract the step text corresponding to each step from the target search text, identify the sentence attribute information of each sentence in the step text, and filter the target sentence from the step text based on the sentence attribute information to obtain the guiding text, and so on.

[0184] When a candidate video contains video or image content, the server extracts guiding keywords from the guiding text and selects at least one image corresponding to the guiding keyword from the candidate video and auxiliary search images to obtain the search condition image. Alternatively, the server can perform feature extraction on the guiding text to obtain guiding text features, extract image features from the candidate video and auxiliary search images to obtain current image features, calculate the feature similarity or feature distance between the guiding text features and the current image features to obtain the image-text relevance. This image-text relevance indicates the degree of association between the guiding text and the images in the candidate video and the auxiliary search images. Based on the image relevance, at least one image is selected from the candidate video and auxiliary search images to obtain the search condition image, and so on. The search condition image and guiding text are used as search condition information.

[0185] When the candidate video does not contain video content or images, the server can filter out at least one image corresponding to the guiding text from the auxiliary search images to obtain the search condition image, and use the search condition image and guiding text as search condition information.

[0186] When the search request type is a short text search request, and the candidate videos contain image content or video content, the server extracts text features from the target search text to obtain search text features. It then filters the current image features from the candidate videos and auxiliary search images, calculates the similarity and feature distance between the search text features and the current image features to obtain feature relevance. Based on the feature relevance, it selects at least one image from the candidate videos and auxiliary search images to obtain a related image. Alternatively, it extracts at least one keyword from the target search text, selects at least one image from the candidate videos and auxiliary search images based on the keyword, and so on. The related image and the target search text are used as search condition information corresponding to the video search request.

[0187] When the search request type is a short text search request and the candidate videos do not contain image content or video content, the server filters out at least one image corresponding to the target search text from the auxiliary search images to obtain the associated image, and uses the associated image and the target search text as the search condition information corresponding to the video search request.

[0188] When the search request type is a long text search request, the server can query the modal information of image modality and table modality. When image modal information exists, at least one content image is extracted from the image modal information. When table modal information exists, table information is extracted from the table modal information.

[0189] The server can use an OCR model to identify at least one text character in the content image and the auxiliary search image to obtain the current text information. Alternatively, it can use a text feature extraction model to extract text features from the content image and the auxiliary search image to obtain the current text features. Based on the current text features, the server can determine the text information in the content image and the auxiliary search image to obtain the current text information, and so on.

[0190] The server can identify the position of the table in the content image or candidate video from the table information, obtain the table position, extract the area corresponding to the table position in the content image or candidate video, and thus obtain at least one table image. Alternatively, it can filter the table template image corresponding to the table information from a preset set of table template images, insert the table information into the table template image, and thus obtain at least one table information. Or, it can also crop at least one image from the content image based on the table information to obtain a table image, and so on.

[0191] The server can use the current text information, content images, table images, and auxiliary search images as search criteria for the video search request.

[0192] When the search criteria information includes guiding text and search criteria image, the server can use the guiding text as the current criteria text and the search criteria image as the current criteria image. When the search criteria information includes related images and target search text, the server can use the target search text as the current criteria text and the related images as the current criteria image. When the search criteria information includes current text information, content images, table images, and auxiliary search images, the server can use the current text information as the current criteria text and the content images, table images, and auxiliary search images as the current criteria images.

[0193] The server uses an image generation model to extract features from the current conditional text to obtain the search text features, and uses the same model to extract features from the current conditional image to obtain the search image features.

[0194] The server can randomly sample noise from a preset noise set to obtain the current noise, or it can perform Gaussian sampling on the noise in the preset noise set to obtain the current noise, and so on.

[0195] The server can add the noise value at at least one pixel location of the current noise to the corresponding pixel value in the preset image to obtain the current noise image. Alternatively, it can extract the noise value at at least one pixel location from the current noise, identify the pixel value at that pixel location in the preset image, and replace the pixel value at that pixel location with the noise value to obtain the current noise image, and so on.

[0196] The server can use an image generation model to denoise the current noisy image based on the search image features and search text features, obtain the denoised image, and then return to execute the step of sampling the current noise from the preset noise set until the preset number of denoising times is reached, thereby obtaining the target search image corresponding to the target search text.

[0197] Optionally, in some embodiments, the server may also train a preset image generation model to obtain an image generation model. The process of the server training the preset image generation model can be as follows:

[0198] The server can obtain the training dataset uploaded by the client or terminal. The training dataset includes at least one sample pair, which includes a text sample and an image sample corresponding to the text sample. Alternatively, it can obtain at least one text sample and select the image sample corresponding to the text sample from a preset image set to obtain at least one sample pair, and so on.

[0199] The server can extract features from text samples to obtain sample text features, extract features from image samples in the training dataset to obtain candidate sample image features, and calculate the feature similarity or feature distance between the sample text features and the candidate sample image features to obtain text similarity. Alternatively, it can extract features from text samples to obtain sample text features, extract features from image samples in the training dataset to obtain candidate sample image features, convert the candidate sample image features into text-dimensional features to obtain converted features, and calculate the feature similarity or feature distance between the sample text features and the converted features to obtain text similarity, and so on.

[0200] The server can filter out at least one image sample whose text similarity exceeds a preset similarity threshold from the image samples, thereby obtaining at least one nearest neighbor image sample. Alternatively, it can sort the image samples based on text similarity to obtain a sorting result. Based on the sorting result of the image samples, it can filter out at least one image sample within a preset sorting range (e.g., TOP N, where N is any positive integer) from the image samples, thereby obtaining at least one nearest neighbor image sample, and so on.

[0201] The server adds neighboring image samples to the sample pairs corresponding to the text samples, thereby obtaining the sample triplet corresponding to the sample pairs.

[0202] The server extracts features from the text samples in the sample triplet to obtain sample text features, and extracts features from the neighboring image samples to obtain sample image features. It then selects target noise from a preset noise set and adds noise to the preset image based on the target noise to obtain a noisy image.

[0203] The server uses a preset image generation model to predict the noise to be added to the noisy image based on the features of the sample text and sample image, obtaining predicted noise data. Based on this predicted noise data, the server denoises the noisy image to obtain the current denoised image. This denoised image is then used as the preset image to return and execute the step of filtering the target noise from the preset noise set until a preset number of denoising steps is reached, thus obtaining the target image corresponding to the text sample.

[0204] The server can extract features from the target image to obtain the target image features, calculate the feature distance or feature similarity between the target image features and the sample images of the image samples, and thus obtain the image loss. Alternatively, it can calculate the pixel difference between the target image and the image samples at each pixel position and fuse the pixel differences to obtain the image loss. Or, it can calculate the squared difference of the pixel values ​​between the target image and the image samples at each pixel position to obtain the image loss, and so on.

[0205] The server can update the network parameters of a preset image generation model using gradient descent algorithm based on image loss, obtaining an updated image generation model. This updated model is then used as the preset image generation model, and the server returns to the step of obtaining the training dataset until the preset image generation model converges, thus obtaining the final image generation model. Alternatively, it can determine the image loss of the sample triples based on the target image and image samples. Based on this image loss, it can use other model parameter update algorithms to update the network parameters of the preset image generation model, obtaining an updated image generation model. This updated model is then used as the preset image generation model, and the server returns to the step of obtaining the training dataset until the preset image generation model converges, thus obtaining the final image generation model, and so on.

[0206] Optionally, in some embodiments, the server may further filter at least one image corresponding to the target search text from the candidate video and auxiliary search images when the candidate video contains image content or video content, to obtain the image to be edited; and when the candidate video does not contain image content or video content, it may filter at least one image corresponding to the target search text from the auxiliary search images to obtain the image to be edited.

[0207] The server can extract semantic features from the target search text to obtain the current search semantic features, extract semantic features from the image to be edited to obtain the current semantic features, compare the current search semantic features and the current semantic features to obtain the initial editing intent information. When the image to be edited is an image in the auxiliary search images, the initial editing intent information is used as the editing intent information. When the image to be edited is not an image other than the auxiliary search images, image features are extracted from the auxiliary search images to obtain the current auxiliary image features. The initial editing intent information is adjusted based on the current auxiliary image features to obtain the editing intent information of the image to be edited.

[0208] The server uses an image editing model to identify the task type and target editing object of the editing task from the editing intent information. Based on the task type and target editing object, it detects or segments the region to be edited in the image to be edited. According to the editing intent information, it generates the editing content corresponding to the editing task and merges the editing content with the original content in the region to be edited to obtain the target search image corresponding to the video search request.

[0209] 207. The server determines the target search content corresponding to the target search request based on the target search text and the target search image.

[0210] For example, the server can filter out the content display frame corresponding to the video search request from a preset set of content display frames, add the target search text and target search image to the content display frame, and thus obtain the target search content. Alternatively, when the target search text includes text for multiple steps, the server can also filter out the image corresponding to each step from the target search images, and combine the target search text and target search images in the order of the steps to obtain the target search content. The target search text can be encoded using a text encoding network in a multimodal large language model to obtain target text features, and the target search image can be encoded using an image encoding network in a multimodal large language model. The process involves image encoding to obtain target image features, feature transformation of target text features and target image features to obtain transformed text features corresponding to the target text features and transformed image features corresponding to the target image features, semantic analysis of transformed text features and transformed image features to obtain text semantic information corresponding to transformed text features and image semantic information corresponding to transformed image features, fusion of text semantic information and image semantic information to obtain multimodal semantic information, generation of modal content of at least one modality based on multimodal semantic information, fusion of modal content to obtain target search content, which includes content of at least one modality.

[0211] Optionally, in some embodiments, the server may also use a graph network to predict the current search content of at least one modality corresponding to the search suggestion information, and generate the target search content corresponding to the video search request based on the current search content, wherein the target search content includes the content of at least one modality.

[0212] Optionally, in some embodiments, the server may display a content display page corresponding to the video search request, which may display the target search content; or, the content display address may be extracted from the video search request, and the target search content may be sent to the display terminal based on the content display address so that the display terminal can display the target search content; or, the target terminal information of the target object may be extracted from the video search request, and the target search content may be sent to the target object's target terminal based on the target terminal information; or, the target search content may be sent to the interactive server of the content platform corresponding to the video search request so that the interactive server can send the target search content to at least one interactive terminal, and so on.

[0213] As can be seen from the above, in this embodiment of the application, after obtaining a video search request, the server filters at least one candidate video from a preset content set based on at least one search hint content carried in the video search request, extracts modal information of at least one modality from the candidate video, then generates target search text corresponding to the video search request based on the modal information, retrieves at least one auxiliary search image based on the search hint content, generates a target search image corresponding to the video search request based on the target search text and the auxiliary search image, and then determines the target search content corresponding to the target search request based on the target search text and the target search image. The target search content includes content of at least one modality. Since this scheme can use the multimodal data of the candidate videos filtered by the search hint content to generate target search text, and then retrieve auxiliary search images through the search hint content to combine with the target search text to jointly generate the target search image, the consistency between the target search text and the target search image can be improved. Moreover, since the auxiliary search image is introduced as additional knowledge in the process of generating the target search image, the possibility of the target search image containing illusions is reduced. Therefore, the accuracy of content search can be improved.

[0214] To better implement the above methods, this application also provides a content search device that can be integrated into an electronic device, such as a server or terminal. The terminal may include a tablet computer, a laptop computer, and / or a personal computer.

[0215] For example, such as Figure 6 As shown, the content search device may include an acquisition unit 301, a filtering unit 302, a retrieval unit 303, a generation unit 304, and a determination unit 305, as follows:

[0216] (1) Obtain unit 301;

[0217] The acquisition unit 301 is used to acquire a content search request, which carries at least one search suggestion.

[0218] (2) Filtering unit 302;

[0219] The filtering unit is used to filter at least one candidate content from a preset content set based on the search prompts, and to extract modal information of at least one modality from the candidate content.

[0220] For example, the filtering unit 302 can be used to identify the search intent corresponding to the content search request in the search prompt information, obtain search intent information, filter at least one preset content in the preset content set based on the search intent information to obtain initial candidate content, segment at least one content fragment in the initial candidate content to obtain at least one candidate content, and extract modal information of at least one modality from the candidate content.

[0221] (3) Retrieval unit 303;

[0222] The retrieval unit 303 is used to generate target search text corresponding to the content search request based on modal information, and retrieve at least one auxiliary search image according to the search prompt content.

[0223] For example, the retrieval unit 303 can be used to classify content search requests according to search needs based on search intent information to obtain the search need type of the content search request. When the search need type is a step list requirement, the target search text corresponding to the content search request is generated based on modal information. When the search need type is a long text search requirement or a short text search requirement, the target search text corresponding to the content search request is generated based on search hint information and modal information. At least one keyword is extracted from the search text, and at least one image related to the keyword is selected from a preset image set to obtain an auxiliary search image.

[0224] (4) Generation unit 304;

[0225] The generation unit 304 is used to generate a target search image corresponding to the content search request based on the target search text and the auxiliary search image.

[0226] For example, the generation unit 304 can be specifically used to: when the search request type is a step list request, identify the guiding text for each step in the target search text, and determine the search condition information corresponding to the content search request based on the guiding text, auxiliary search image, and candidate content; when the search request type is a short text search request, determine the search condition information corresponding to the content search request based on the target search text, auxiliary search image, and candidate content; when the search request type is a long text search request, determine the search condition information corresponding to the content search request based on the candidate content and auxiliary search image; extract the current condition text and current condition image from the search condition information; use an image generation model to extract features from the current condition text to obtain search text features; use an image generation model to extract features from the current condition image to obtain search image features; sample the current noise from a preset noise set; and add noise to the preset image based on the current noise to obtain a current noise image; and use an image generation model to denoise the current noise image based on the search image features and search text features to obtain the target search image corresponding to the target search text.

[0227] (5) Determine unit 305;

[0228] The determining unit 305 is used to determine the target search content corresponding to the target search request based on the target search text and the target search image, wherein the target search content includes content of at least one modality.

[0229] For example, the determining unit 305 can be used to fuse the target search text and the target search image to obtain the target search content corresponding to the content search request, or, based on the target search text and the target search image, to generate the target search content corresponding to the content search request using a multimodal large language model, wherein the target search content includes content of at least one modality.

[0230] Optionally, in some embodiments, the content search device may further include a training unit 306, such as... Figure 7 As shown, the specific details are as follows:

[0231] Training unit 306 can be used to train a preset image generation model to obtain an image generation model.

[0232] For example, training unit 306 can be used to obtain a training dataset, which includes at least one sample pair, which includes a text sample and an image sample corresponding to the text sample. It can construct a sample triplet corresponding to the sample pair, which includes the sample pair and at least one nearest neighbor image sample corresponding to the text sample in the sample pair. Based on the sample triplet, a preset image generation model is trained to obtain the image generation model.

[0233] Optionally, in some embodiments, the content search device may further include a search unit 307, such as... Figure 8 As shown, the specific details are as follows:

[0234] Search unit 307 is used to generate target search content corresponding to search prompts using a graph network, wherein the target search content includes content of at least one modality.

[0235] For example, the search unit 307 can be used to predict the current search content of at least one modality corresponding to the search prompt content using a graph network. The graph network is constructed by the search prompt content sample and the search content sample of at least one modality corresponding to the search prompt content sample. Based on the current search content, the target search content corresponding to the content search request is generated. The target search content includes content of at least one modality.

[0236] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0237] As can be seen from the above, in this embodiment of the application, after the acquisition unit 301 acquires the content search request, the filtering unit 302 filters at least one candidate content from a preset content set based on at least one search hint content carried in the content search request, and extracts modal information of at least one modality from the candidate content. Then, the retrieval unit 303 generates the target search text corresponding to the content search request based on the modal information, and retrieves at least one auxiliary search image based on the search hint content. The generation unit 304 generates the target search image corresponding to the content search request based on the target search text and the auxiliary search image. Finally, the determination unit 305 determines the target search image based on the target search text. The method uses a target search image to determine the target search content corresponding to the target search request. This target search content includes content in at least one modality. Since this method can generate target search text using multimodal data of candidate content filtered by search suggestions, and then retrieve auxiliary search images from the search suggestions to combine with the target search text to generate a target search image, the consistency between the target search text and the target search image can be improved. Moreover, since auxiliary search images are introduced as additional knowledge in the process of generating the target search image, the possibility of the target search image containing illusions is reduced, thus improving the accuracy of content search.

[0238] This application also provides an electronic device, such as... Figure 9 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0239] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 9 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0240] The processor 401 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes software programs and / or modules stored in the memory 402, and calls data stored in the memory 402, to perform various functions and process data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0241] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0242] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0243] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0244] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402 to realize various functions, as follows:

[0245] A content search request is obtained, which carries at least one search suggestion. Based on the search suggestion, at least one candidate content is selected from a preset content set, and modal information of at least one modality is extracted from the candidate content. Based on the modal information, target search text corresponding to the content search request is generated, and at least one auxiliary search image is retrieved based on the search suggestion. Based on the target search text and the auxiliary search image, target search image corresponding to the content search request is generated. Based on the target search text and the target search image, target search content corresponding to the target search request is determined, and the target search content includes content of at least one modality.

[0246] For example, an electronic device can acquire a content search request, which carries at least one search suggestion. The search intent corresponding to the content search request is identified from the search suggestion information to obtain search intent information. Based on the search intent information, at least one preset content is selected from a preset content set to obtain initial candidate content. At least one content segment is segmented from the initial candidate content to obtain at least one candidate content, and modal information of at least one modality is extracted from the candidate content. Based on the search intent information, the content search request is classified according to search needs to obtain the search need type of the content search request. When the search need type is a step list need, the target search text corresponding to the content search request is generated based on the modal information. When the search need type is a long text search need or a short text search need, the target search text corresponding to the content search request is generated based on the search suggestion information and modal information. At least one keyword is extracted from the search text, and at least one image related to the keyword is selected from a preset image set to obtain an auxiliary search image. When the search request type is a step list request, the guiding text for each step is identified in the target search text. Based on the guiding text, auxiliary search image, and candidate content, the search condition information corresponding to the content search request is determined. When the search request type is a short text search request, the search condition information corresponding to the content search request is determined based on the target search text, auxiliary search image, and candidate content. When the search request type is a long text search request, the search condition information corresponding to the content search request is determined based on the candidate content and auxiliary search image. The current condition text and current condition image are extracted from the search condition information. An image generation model is used to extract features from the current condition text to obtain search text features. An image generation model is also used to extract features from the current condition image to obtain search image features. Current noise is sampled from a preset noise set, and noise is added to the preset image based on the current noise to obtain a current noise image. Based on the search image features and search text features, an image generation model is used to denoise the current noise image to obtain the target search image corresponding to the target search text. The target search text and target search image are fused to obtain the target search content corresponding to the content search request. Alternatively, based on the target search text and target search image, a multimodal large language model is used to generate the target search content corresponding to the content search request, which includes content in at least one modality.

[0247] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0248] As can be seen from the above, after obtaining a content search request, this embodiment of the application filters at least one candidate content from a preset content set based on at least one search suggestion carried in the content search request, extracts modal information of at least one modality from the candidate content, then generates target search text corresponding to the content search request based on the modal information, retrieves at least one auxiliary search image based on the search suggestion, generates a target search image corresponding to the content search request based on the target search text and the auxiliary search image, and then determines the target search content corresponding to the target search request based on the target search text and the target search image. The target search content includes content of at least one modality. Since this scheme can use the multimodal data of the candidate content filtered by the search suggestion to generate target search text, and then retrieve auxiliary search images through the search suggestion to combine with the target search text to jointly generate the target search image, the consistency between the target search text and the target search image can be improved. Moreover, since the auxiliary search image is introduced as additional knowledge in the process of generating the target search image, the possibility of the target search image containing illusions is reduced, thus improving the accuracy of content search.

[0249] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0250] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the content search methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0251] A content search request is obtained, which carries at least one search suggestion. Based on the search suggestion, at least one candidate content is selected from a preset content set, and modal information of at least one modality is extracted from the candidate content. Based on the modal information, target search text corresponding to the content search request is generated, and at least one auxiliary search image is retrieved based on the search suggestion. Based on the target search text and the auxiliary search image, target search image corresponding to the content search request is generated. Based on the target search text and the target search image, target search content corresponding to the target search request is determined, and the target search content includes content of at least one modality.

[0252] For example, a content search request is obtained, carrying at least one search suggestion. The search intent corresponding to the content search request is identified from the search suggestion information to obtain search intent information. Based on the search intent information, at least one preset content is selected from a preset content set to obtain initial candidate content. At least one content segment is segmented from the initial candidate content to obtain at least one candidate content, and modal information of at least one modality is extracted from the candidate content. Based on the search intent information, the content search request is classified according to search needs to obtain the search need type of the content search request. When the search need type is a step list need, the target search text corresponding to the content search request is generated based on the modal information. When the search need type is a long text search need or a short text search need, the target search text corresponding to the content search request is generated based on the search suggestion information and modal information. At least one keyword is extracted from the search text, and at least one image related to the keyword is selected from a preset image set to obtain auxiliary search images. When the search request type is a step list request, the guiding text for each step is identified in the target search text. Based on the guiding text, auxiliary search image, and candidate content, the search condition information corresponding to the content search request is determined. When the search request type is a short text search request, the search condition information corresponding to the content search request is determined based on the target search text, auxiliary search image, and candidate content. When the search request type is a long text search request, the search condition information corresponding to the content search request is determined based on the candidate content and auxiliary search image. The current condition text and current condition image are extracted from the search condition information. An image generation model is used to extract features from the current condition text to obtain search text features. An image generation model is also used to extract features from the current condition image to obtain search image features. Current noise is sampled from a preset noise set, and noise is added to the preset image based on the current noise to obtain a current noise image. Based on the search image features and search text features, an image generation model is used to denoise the current noise image to obtain the target search image corresponding to the target search text. The target search text and target search image are fused to obtain the target search content corresponding to the content search request. Alternatively, based on the target search text and target search image, a multimodal large language model is used to generate the target search content corresponding to the content search request, which includes content in at least one modality.

[0253] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0254] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0255] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the content search methods provided in the embodiments of this application, the beneficial effects that any of the content search methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0256] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in the various alternative implementations of the content search or user question-and-answer aspects described above.

[0257] The foregoing has provided a detailed description of a content search method and related equipment provided in the embodiments of this application. These related equipment may include content search devices, electronic devices, computer program products, and computer-readable storage media. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are merely for the purpose of helping to understand the method and core ideas of this invention. Furthermore, those skilled in the art will recognize that, based on the ideas of this invention, there may be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A content search method, characterized in that, include: Obtain a content search request, wherein the content search request carries at least one search suggestion; Based on the search suggestions, at least one candidate content is selected from the preset content set, and modal information of at least one modality is extracted from the candidate content. Based on the modal information, the target search text corresponding to the content search request is generated, and at least one auxiliary search image is retrieved according to the search prompt content; Based on the target search text and the auxiliary search image, generate the target search image corresponding to the content search request; Based on the target search text and the target search image, the target search content corresponding to the target search request is determined, and the target search content includes content in at least one modality.

2. The content search method according to claim 1, characterized in that, The step of filtering at least one candidate content from a preset content set based on the search suggestions includes: The search intent corresponding to the content search request is identified from the search suggestion content to obtain search intent information; Based on the search intent information, at least one preset content is selected from the preset content set to obtain initial candidate content; At least one content segment is segmented from the initial candidate content to obtain at least one candidate content.

3. The content search method according to claim 2, characterized in that, The step of filtering at least one preset content from a preset content set based on the search intent information to obtain initial candidate content includes: Filter at least one initial content corresponding to the search intent information from the preset content set; Obtain historical search information of the target object corresponding to the content search request, and determine the content relevance parameter between the initial content and the content search request based on the historical search information; Based on the content relevance parameters, at least one preset content is selected from the initial content to obtain initial candidate content.

4. The content search method according to claim 2, characterized in that, The step of segmenting at least one content fragment from the initial candidate content to obtain at least one candidate content includes: The initial candidate content is sliced ​​to obtain at least one candidate content fragment; Feature extraction is performed on the candidate content fragments, and semantic analysis is performed on the candidate content fragments based on the extracted content features to obtain the semantic information of the candidate content fragments; Based on the semantic information, at least one candidate content segment is selected from the candidate content segments to obtain candidate content.

5. The content search method according to claim 2, characterized in that, The step of generating the target search text corresponding to the content search request based on the modal information includes: Based on the search intent information, the content search requests are classified according to search needs to obtain the search need types of the content search requests; When the search request type is a step list request, the target search text corresponding to the content search request is generated based on the modal information; When the search request type is a long text search request or a short text search request, the target search text corresponding to the content search request is generated based on the search suggestion content and modal information.

6. The content search method according to claim 5, characterized in that, The step of generating the target search image corresponding to the content search request based on the target search text and the auxiliary search image includes: Based on the search request type and the auxiliary search image, determine the search condition information corresponding to the content search request; Based on the search criteria information, at least one image corresponding to the target search text is generated using an image generation model to obtain the target search image.

7. The content search method according to claim 6, characterized in that, The step of determining the search condition information corresponding to the content search request based on the search request type and auxiliary search image includes: When the search request type is the step list request, the guiding text for each step is identified in the target search text, and the search condition information corresponding to the content search request is determined based on the guiding text, auxiliary search image and candidate content. When the search request type is the short text search request, the search condition information corresponding to the content search request is determined based on the target search text, auxiliary search image and candidate content; When the search request type is the long text search request, the search condition information corresponding to the content search request is determined based on the candidate content and the auxiliary search image.

8. The content search method according to claim 7, characterized in that, The step of determining the search condition information corresponding to the content search request based on the guiding text, auxiliary search image, and candidate content includes: When the candidate content includes video content or image content, at least one image corresponding to the guiding text is selected from the candidate content and auxiliary search images to obtain the search condition image, and the search condition image and the guiding text are used as the search condition information. When the candidate content does not contain the video content or image content, at least one image corresponding to the guiding text is selected from the auxiliary search images to obtain the search condition image, and the search condition image and the guiding text are used as the search condition information.

9. The content search method according to claim 7, characterized in that, The step of determining the search condition information corresponding to the content search request based on the target search text, auxiliary search image, and candidate content includes: When the candidate content contains image content or video content, at least one image corresponding to the target search text is selected from the candidate content and auxiliary search images to obtain the associated image, and the associated image and the target search text are used as the search condition information corresponding to the content search request. When the candidate content does not contain the image content or video content, at least one image corresponding to the target search text is selected from the auxiliary search images to obtain the associated image, and the associated image and the target search text are used as the search condition information corresponding to the content search request.

10. The content search method according to claim 7, characterized in that, The step of determining the search condition information corresponding to the content search request based on the candidate content and the auxiliary search image includes: Extract at least one content image and table information from the modal information of the candidate content; Text is extracted from the content image and the auxiliary search image to obtain the current text information, and at least one table image is generated based on the table information; The current text information, content image, table image, and auxiliary search image are used as the search criteria information corresponding to the content search request.

11. The content search method according to claim 6, characterized in that, Before generating at least one image corresponding to the target search text using an image generation model based on the search condition information, and before obtaining the target search image, the process further includes: Obtain a training dataset, which includes at least one sample pair, the sample pair including a text sample and an image sample corresponding to the text sample; Construct sample triples corresponding to the sample pairs, wherein the sample triples include the sample pair and at least one nearest neighbor image sample corresponding to the text sample in the sample pair; Based on the sample triples, the preset image generation model is trained to obtain the image generation model.

12. The content search method according to claim 11, characterized in that, The construction of the sample triplet corresponding to the sample pair includes: Calculate the text similarity between the text sample and the image samples in the training dataset; Based on the text similarity, at least one nearest neighbor image sample corresponding to the text sample is selected from the image samples; The nearest neighbor image sample is added to the sample pair corresponding to the text sample to obtain the sample triplet corresponding to the sample pair.

13. The content search method according to claim 11, characterized in that, The step of training a preset image generation model based on the sample triples to obtain the image generation model includes: Feature extraction is performed on the text samples in the sample triplet to obtain sample text features, and feature extraction is performed on the neighboring image samples to obtain sample image features; Target noise is selected from a preset noise set, and noise is added to a preset image based on the target noise to obtain a noisy image; Based on the sample text features and sample image features, the preset image generation model is used to denoise the noisy image to obtain the target image corresponding to the text sample; Based on the target image and image samples, the preset image generation model is converged to obtain the image generation model.

14. The content search method according to claim 1, characterized in that, The step of retrieving at least one auxiliary search image based on the search suggestion content includes: Text information is extracted from the search suggestion content to obtain the search text corresponding to the search suggestion content; When the candidate content includes video content or image content, at least one image corresponding to the search text is selected from the preset image set and the candidate content to obtain an auxiliary search image; When the candidate content does not contain video or image content, at least one image corresponding to the search text is selected from the preset image set to obtain an auxiliary search image.

15. The content search method according to claim 1, characterized in that, The step of generating the target search image corresponding to the content search request based on the target search text and the auxiliary search image includes: Select at least one image to be edited from the candidate content and auxiliary search images; Based on the target search text and the auxiliary search image, determine the editing intent information of the image to be edited; Based on the editing intent information, the image to be edited is edited to obtain the target search image corresponding to the content search request.

16. The content search method according to claim 1, characterized in that, Following the content search request, the process also includes: A graph network is used to predict the current search content of at least one modality corresponding to the search suggestion content. The graph network is constructed using search suggestion content samples and search content samples of at least one modality corresponding to the search suggestion content samples. Based on the current search content, generate the target search content corresponding to the content search request, wherein the target search content includes content in at least one modality.

17. A content search device, characterized in that, include: An acquisition unit is used to acquire a content search request, wherein the content search request carries at least one search suggestion content; The filtering unit is used to filter at least one candidate content from a preset content set based on the search prompt content, and extract modal information of at least one modality from the candidate content. The retrieval unit is configured to generate target search text corresponding to the content search request based on the modal information, and retrieve at least one auxiliary search image according to the search prompt content; The generation unit is used to generate a target search image corresponding to the content search request based on the target search text and the auxiliary search image; The determining unit is configured to determine the target search content corresponding to the target search request based on the target search text and the target search image, wherein the target search content includes content in at least one modality.

18. An electronic device, characterized in that, It includes a processor and a memory, the memory storing an application program, and the processor running the application program within the memory to perform the steps of the content search method according to any one of claims 1 to 16.

19. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the steps of the content search method according to any one of claims 1 to 16.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the content search method according to any one of claims 1 to 16.

Images