Image search method, device, intelligent agent, electronic device and storage medium

Abstract

The present disclosure provides an image search method, apparatus, intelligent agent, electronic device, and storage medium, relating to the field of artificial intelligence technology, particularly computer vision, deep learning, large models, image search, and other technical fields, and can be applied to scenarios such as AIGC-based content generation based on artificial intelligence. The method includes: obtaining at least one first candidate image that matches input text information; using a first large model to perform semantic analysis on the input text information to generate at least one question-answer pair, the question-answer pair including question information and first answer information; using a second large model to perform graphic analysis on the at least one question information and at least one first candidate image to generate second answer information for each question information; and determining, based on a comparison result between the at least one first answer information and the at least one second answer information, at least one target image that matches the image search requirement from the at least one first candidate image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, particularly computer vision, deep learning, large models, image search, and other technical fields, and can be applied to scenarios such as AIGC-based content generation based on artificial intelligence. Specifically, it relates to an image search method, device, intelligent agent, electronic device, and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, large model technology has also been applied in various fields. For example, large models are used to achieve image search.

[0003] However, there are currently some deviations in the details between the images obtained by image search based on large models and the input information, such as semantic inconsistency. Summary of the Invention

[0004] The present disclosure provides an image search method, device, intelligent agent, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, an image search method is provided, including: obtaining at least one first candidate image that matches input text information, wherein the input text information is used to represent image search requirements; performing semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; performing graphic-text analysis on at least one question information and at least one first candidate image using a second large model to generate second answer information for answering each question information; and determining at least one target image that matches the image search requirement from the at least one first candidate image based on a comparison result between the at least one first answer information and the at least one second answer information.

[0006] According to another aspect of the present disclosure, an image search device is provided, including: an acquisition module for acquiring at least one first candidate image similar to input text information input by a user, wherein the input text information is used to represent the user's image search needs; a semantic analysis module for performing semantic analysis on the input text information using a first large model, and generating at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; a graphic-text analysis module for performing graphic-text analysis on at least one question information and at least one first candidate image using a second large model, and generating second answer information for answering each question information; and a determination module for determining at least one target image that matches the image search needs from at least one first candidate image based on a comparison result between at least one first answer information and at least one second answer information.

[0007] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the method provided by the embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements the above method when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0013] Figure 1 Schematically illustrates an exemplary system architecture to which the image search method and apparatus according to an embodiment of the present disclosure can be applied;

[0014] Figure 2 The following schematically shows a flow chart of an image search method according to an embodiment of the present disclosure;

[0015] Figure 3 A schematic diagram schematically illustrates a scenario of an image search method according to an embodiment of the present disclosure;

[0016] Figure 4 The following schematically illustrates a scenario diagram for generating question-answer pairs according to an embodiment of the present disclosure;

[0017] Figure 5 A schematic diagram of a scenario for obtaining first prompt information according to an embodiment of the present disclosure is shown schematically;

[0018] Figure 6 A schematic diagram of a scenario for determining a target image according to an embodiment of the present disclosure is shown schematically;

[0019] Figure 7A A schematic diagram of a scenario in which a target image is determined using the third largest model according to an embodiment of the present disclosure is shown;

[0020] Figure 7B A schematic diagram of a scene for determining a target image using a third large model according to another embodiment of the present disclosure is shown schematically;

[0021] Figure 8 Schematically illustrates a scenario diagram of the third model performing graphic and text analysis tasks and interpretation tasks according to an embodiment of the present disclosure;

[0022] Figure 9 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown;

[0023] Figure 10 A block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown;

[0024] Figure 11 A block diagram schematically illustrates a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0025] Figure 12 A block diagram of an electronic device 1200 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0027] Although existing image search solutions can search and return images based on text information entered by users, these returned images may not meet or fully meet the search intent. For example, the returned images may not meet or fully meet the user's needs at the semantic level.

[0028] For example, content-based image search performs matching by extracting low-level features of the image, such as color histograms and edge features, while label-based image search utilizes metadata or manually annotated labels in the image for search. These methods can only process surface features of the image and cannot understand or capture deep semantic features. The above methods are only capable of simpler image matching tasks. For complex language descriptions or combined search tasks, the search accuracy is low. In addition, image search based on visual language models can map data from visual and language modalities to a shared vector space to achieve basic semantic alignment, such as calculating the similarity between visual modality vectors and language modality vectors. However, semantic alignment in vector space alone is relatively rough and cannot ensure precise semantic consistency. For example, subtle image details (such as color, shape, position) or complex semantic relationships (such as interactions between objects) may not be accurately captured and matched.

[0029] In summary, there are semantic deviations in details and complex semantic relationships between the images obtained by existing image search schemes and the input information, and the image search accuracy is low.

[0030] In order to at least partially solve the above technical problems, the embodiments of the present disclosure provide an image search method, comprising: obtaining at least one first candidate image that matches input text information, wherein the input text information is used to represent the image search requirements; using a first large model to perform semantic analysis on the input text information to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; using a second large model to perform graphic analysis on at least one question information and at least one first candidate image to generate second answer information for answering each question information; and based on the comparison result between at least one first answer information and at least one second answer information, determining at least one target image that matches the image search requirements from at least one first candidate image. Therefore, the embodiments of the present disclosure can at least solve the problem of semantic deviation between the output image and the input, and achieve the technical effect of improving the accuracy of image search.

[0031] Figure 1 An exemplary system architecture to which the image search method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0032] It should be noted that Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure. This does not mean that the embodiments of the present disclosure cannot be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the image search method and apparatus may be applied may include a terminal device, but the terminal device may implement the method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] A user may use a first terminal device 101, a second terminal device 102, or a third terminal device 103 to interact with a server 105 via a network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, or the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0035] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal devices.

[0037] The server can be a cloud server, also known as a cloud computing server or cloud host. It is a hosting product within the cloud computing service system that addresses the management difficulties and poor scalability of traditional physical hosts and VPS services ("Virtual Private Servers" or "VPS"). The server can also be a distributed system server or a server integrated with blockchain.

[0038] It should be noted that the image search method provided in the embodiments of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the image search apparatus provided in the embodiments of the present disclosure can also be provided in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0039] Alternatively, the image search method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the image search apparatus provided in the embodiment of the present disclosure may generally be provided in the server 105. The image search method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image search apparatus provided in the embodiment of the present disclosure may also be provided in a server or server cluster that is different from the server 105 and that is capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] For example, a user inputs input text information for image search through the first terminal device 101, the second terminal device 102, and the third terminal device 103. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can obtain at least one first candidate image that matches the input text information, wherein the input text information is used to represent the image search requirements; the input text information is semantically analyzed using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; the at least one question information and the at least one first candidate image are graphically analyzed using a second large model to generate second answer information for answering each question information; and based on the comparison result between the at least one first answer information and the at least one second answer information, at least one target image that matches the image search requirements is determined from the at least one first candidate image.

[0041] Alternatively, a document is sent to the server 105 through the first terminal device 101, the second terminal device 102, and the third terminal device 103 to obtain input text information for image search, and the above-mentioned image search method is performed with the help of the server 105 to determine and return at least one searched target image to the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0043] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0044] In the technical solution disclosed herein, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0045] Figure 2 The flowchart of the image search method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0047] In operation S210 , at least one first candidate image matching input text information is acquired, where the input text information is used to represent an image search requirement.

[0048] Input text information includes text information obtained through at least one user input operation, for example, text information obtained by combining at least one conversation to clarify image search requirements. Alternatively, input text information may be text information obtained by processing multimodal input information, such as converting information from visual or audio modalities into input text information from a language modality.

[0049] In an application scenario, users can communicate with digital humans / virtual humans through 2D perspectives such as display interfaces and 3D perspectives such as virtual augmented reality to input multimodal information, thereby directly obtaining text information in the language modality, or converting information in other modalities into text information in the language modality to obtain input text information.

[0050] Image search needs, also known as image search intent, include various requirements related to image search tasks, such as searching for images similar to a specific image, searching for images of a certain category, or searching for images of a specific product.

[0051] The first candidate image may be an image that matches the input text information, obtained through a pre-screening operation. For example, a method for obtaining at least one first candidate image that matches the input text information may include: screening out images of the same type, size, and source as the input text information from multiple images to obtain the first candidate image. Alternatively, it may be an image obtained after a rough screening at the semantic level; for example, semantically aligning the input text information and multiple images in a vector space, and screening out images that are similar to the input text information, and using the similar images as the first candidate images.

[0052] In operation S220, semantic analysis is performed on the input text information using the first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information.

[0053] The first large model can be a pre-trained single-modal large model, such as a large language model (LLMs) of the language modality; it can also be a pre-trained multimodal large model (Large Multimodal Models, LMMs) that can process the language modality.

[0054] The first large model can perform semantic analysis on the input text information, serving as model input, and output at least one question-answer pair. Specifically, the semantic analysis can be performed by splitting / extracting information from the input text information along a semantic dimension, and outputting a pair of question information and first answer information. Each question-answer pair output by the first large model represents the result of a local semantic analysis of the input text information.

[0055] In an exemplary embodiment, the question information in the question-answer pair contains less information than the input text information, such as the number of objects and the number of attributes of the objects contained are less than those in the input text information.

[0056] For example, the input text information is "Person 1 is eating at the dining table", and the question-answer pairs output by the first model include: "Is Person 1 eating? Yes", "Is the food on the dining table? Yes", "Is Person 1 at the dining table? Yes", and the question information is "Is Person 1 eating?", "Is the food on the dining table?", "Is Person 1 at the dining table?", and the first answer information is "Yes", "Yes", and "Yes" respectively.

[0057] In operation S230, a second large model is used to perform graphic and text analysis on at least one question information and at least one first candidate image to generate second answer information for answering each question information.

[0058] The second largest model is a multimodal large model LMMs capable of processing visual and language modalities. The second largest model is a pre-trained multimodal large model. The input of the second largest model is multimodal information, such as at least one question information and at least one candidate image.

[0059] The second model performs a text-image analysis on at least one question message and at least one candidate image through a text-image analysis task. The text-image analysis task may be to understand the information in the first candidate image and answer the question message.

[0060] In one exemplary embodiment, the second large model can perform an image-text analysis task based on a single question and a single first candidate image to generate a second answer to the question. Alternatively, the second large model can also perform an image-text analysis task based on multiple questions and a single first candidate image, generating a second answer to each question after understanding the content of the first candidate image. For example, multiple questions and a single first candidate image can be input into the second large model in an array format.

[0061] For example, taking question information 1 as "Is person 1 at the dinner table?", after the second largest model understands the first candidate image 1, it generates the second answer information to answer the above question information 1 as "no"; after understanding the first candidate image 2, it generates the second answer information to answer the above question information 1 as "yes".

[0062] In operation S240, at least one target image matching the image search requirement is determined from the at least one first candidate image based on a comparison result between the at least one first answer information and the at least one second answer information.

[0063] The first answer information and the second answer information are identical in number and represent two pieces of answer information from the first and second largest models for the same question information. Thus, the first and second answer information for the same question information can be compared to select at least one target image that matches the image search requirement from the at least one first candidate image.

[0064] For example, for at least one question information, the comparison results between the first answer information and the second answer information for each question information can be combined to screen out at least one target image. The comparison result can be the similarity between the first answer information and the second answer information. For example, at least one target image can be screened out by combining the similarity between the first answer information and the second answer information for each question information. For another example, the similarity between the first answer information and the second answer information for each question information can be weighted and summed, and the target image can be screened out based on the summed result. Alternatively, the first answer information and the second answer information for each question can be determined based on whether they are the same and the number of similarities or differences can be counted to screen out at least one target image. In addition, the number of target images can be determined based on a predetermined threshold or screening number.

[0065] In the embodiment of the present disclosure, since the first large model splits the input text information into at least one question-answer pair, the question information in the question-answer pair is more fine-grained than the semantic granularity of the input text information, and the first answer information and the second answer information are also responses to the more fine-grained question information. Therefore, based on the comparison results of the first answer information and the second answer information, it is possible to screen at a more fine-grained local semantic perspective, thereby selecting a target image with a smaller local semantic deviation from the first candidate image, and the image search accuracy is higher. In addition, since the comparison results of the first answer information and the second answer information under at least one question information are integrated, the embodiment of the present disclosure can also ensure a target image with a smaller overall semantic deviation, and the image search accuracy is higher.

[0066] In addition, before using the second largest model for image and text analysis, a preliminary screening is performed to obtain the first candidate image that matches the input text information. This ensures the accuracy of the image search while reducing the amount of data processed by the second largest model, making the image search faster.

[0067] According to an embodiment of the present disclosure, for operation S220, using the first large model to perform semantic analysis on the input text information to generate at least one question-answer pair includes: obtaining first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; using the first large model to perform a semantic analysis task based on the first prompt information and the input text information to extract at least one sub-input text information from the input text information; and using the first large model to perform a form conversion task on each sub-input text information based on the first prompt information to convert the sub-input text information into a question-answer pair.

[0068] The first large model is used to sequentially perform the semantic analysis task and the form conversion task. For example, a pre-built first prompt message can be used to prompt the first large model to sequentially perform the above two tasks. The first prompt message can be directly obtained when using the first large model.

[0069] The first prompt information reserves a space for filling in the input text information. For example, the first prompt information includes a placeholder [[INSTRUCTION]]. After obtaining the first prompt information, the current input text information can be filled in the placeholder [[INSTRUCTION]] to obtain a new input.

[0070] The semantic analysis task, i.e., the first model, understands and analyzes the semantics of the input text information and extracts at least one sub-input text information from the input text information according to the semantics. Each sub-input text information includes part of the semantics of the input text information.

[0071] The sub-input text information can be in the form of a declarative sentence. The form of the sub-input text information can be specified in the first prompt information. For example, the information in the first prompt information prompting the first large model to perform the semantic analysis task can be "decompose the input text instruction into multiple verifiable declarative sentences," where "text instruction" refers to the input text information and "declarative sentence" refers to the form of the sub-input information.

[0072] In an exemplary embodiment, the sub-input text information may be directly decomposed from the input text information, or may be obtained by extracting information from the input text information and then recombining it.

[0073] The form conversion task is used to convert the sub-input text information obtained by the semantic analysis task into a predetermined form, such as a question-answer pair.

[0074] In an exemplary implementation, the first prompt information may be used to limit the form of the first answer information so that it is the same as or similar to the form of the second answer information, thereby facilitating subsequent comparison of the first answer information and the second answer information.

[0075] For ease of understanding, an example of the first prompt information is given below. For example, the first prompt information can be: "#Task description: The semantic analysis task involves decomposing a text instruction into multiple simple and verifiable propositions, and the unique answer to each proposition is "yes" or "no". According to the provided instructions, you need to decompose the instruction into several atomic propositions and corresponding answers, following the two steps below. ## Step 1: Declarative sentence conversion: Decompose the input text instruction into multiple verifiable declarative sentences. ## Step 2: Question form conversion: Convert each declarative sentence into a question form and provide the correct answer according to the given instructions. Text instructions will be provided to you to complete this task: Enter text information." Steps 1 and 2 correspond to the semantic analysis task and the form conversion task, respectively.

[0076] In an embodiment of the present disclosure, a first prompt message is used to cause the first large model to sequentially perform a semantic analysis task and a form conversion task to obtain at least one question-answer pair. This hierarchical sequential task execution method enables the first large model to better understand the complex semantic relationships and detailed information in the input text information, thereby improving the accuracy of local semantic extraction of the question information in the question-answer pair. In addition, converting the sub-input text information into a question-answer pair format facilitates subsequent graphic and text analysis of the question information, as well as comparison of the second answer information with the first answer information.

[0077] Figure 3 The following schematically shows a scene diagram of an image search method according to an embodiment of the present disclosure. Figure 3As shown, embodiment 300 is used to search for at least one target image that matches the image search requirement represented by input text information 301 , such as target image 1 306 - 1 . . .

[0078] In this embodiment, input text information 301 and first prompt information are input into a first large model M1. The first large model M1 can perform a semantic analysis task T1 based on the first prompt information and the input text information 301 to obtain at least one sub-input text information, such as sub-input text information 1...sub-input text information M. The first large model M1 can then perform a form conversion task T2 on each sub-input text information. For example, the first large model M1 performs form conversion task T2 on sub-input text information 1 to obtain question-answer pair 1 303-1, which includes question information 303-11 and first answer information 303-12.

[0079] In addition, for the input text information 301, at least one first candidate image matching the input text information 301 may be screened out, such as first candidate image 1 302-1 . . . first candidate image N 302-N.

[0080] After the first large model M1 outputs question information 303-11, question information 303-11 and first candidate image 1 302-1 can be input into the second large model M2. After understanding first candidate image 1 302-1, the second large model M2 generates second answer information 304-11 to answer question information 303-11. After comparing the first answer information 303-12 of question information 303-11 with the second answer information 304-11, a comparison result 305-1 is obtained. Similarly, the process of obtaining comparison results for other question information and other first candidate images is as described above and will not be repeated here.

[0081] After obtaining a comparison result between the at least one first answer information and the at least one second answer information, the comparison result can be used to select at least one target image from the at least one first candidate image. For example, based on the comparison results 305-1..., it is determined whether to select the first candidate image 301-1 as the target image from the first candidate images 1 302-1...first candidate images N 302-N.

[0082] According to an embodiment of the present disclosure, a semantic analysis task is performed based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information, including: performing a semantic analysis task based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; wherein the semantic dimension includes at least one of the following: color, shape, and background.

[0083] The input text information may include at least one object, such as a person, animal, plant, or object. In some embodiments, the first large model may extract at least one sub-input text information from the input text information based on a multi-object perspective, such as understanding complex semantic relationships between objects. To better understand local semantics from multiple semantic dimensions, semantic dimension definitions may be added to the first prompt information, enabling the first large model to extract at least one sub-input text information from multiple semantic dimensions.

[0084] For example, the first prompt information may include a prompt of a semantic dimension, such as the first prompt information may include "decompose the input text instruction into multiple verifiable declarative sentences, and the multiple declarative sentences cover at least one of the following semantic dimensions: color, shape, background."

[0085] Alternatively, the first prompt information may also provide a semantic analysis example to help the first large model understand how to extract at least one sub-input text information from the input text information based on multiple semantic dimensions.

[0086] For example, the first prompt information A may be "decompose the input text instruction into multiple verifiable declarative sentences. The following is an example for reference: For the text instruction - a woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her. The decomposed declarative sentences are: There is a lady in a black shirt in the picture. The lady in the picture is cooking. There is a black dog in the picture. The black dog in the picture is next to the lady. Please query according to ##: Execute text instruction: [[INSTRUCTION]]" It can be understood that the sub-text input information in the example includes the objects "woman" and "dog", the colors include "black shirt" and "black dog", and the background includes "black dog next to the lady".

[0087] In an embodiment of the present disclosure, since at least one sub-input text information is based on multiple semantic dimensions, the semantic dimensions of the question information obtained after converting the sub-input text information are richer, and can focus on complex semantic relationships and detailed information, thereby improving the accuracy of the question information and helping to improve the accuracy of subsequent image searches.

[0088] Figure 4 The following schematically shows a scenario diagram of generating question-answer pairs according to an embodiment of the present disclosure. Figure 4As shown, in embodiment 400, the first large model M1 performs semantic analysis task T1 based on input text information 401 and first prompt information, obtaining multiple sub-input text information for multiple objects. For example, object 1 402-1 includes sub-input text information 1 402-11 under semantic dimension 1... and sub-input text information a 402-1a under semantic dimension a; object 2 402-2 includes sub-input text information...

[0089] For each semantic dimension, the first large model M1 can be used to perform a form conversion task T2 to output a question-answer pair. For example, for sub-input text information 1 402-11, performing form conversion task T2 produces question-answer pair 1 403-1, where question-answer pair 1 403-1 includes question information 403-11 and first answer information 403-12.

[0090] According to an embodiment of the present disclosure, obtaining the first prompt information includes: combining a semantic splitting example and a form conversion example to obtain the first prompt information.

[0091] Examples of semantic splitting include: at least one sub-input text information obtained after performing a semantic splitting task based on predetermined input text information, or at least one sub-input text information of multiple semantic dimensions.

[0092] Examples of form conversion include: question-answer pairs obtained after performing a form conversion task based on predetermined sub-input text information.

[0093] The semantic splitting example and the form conversion example are the results obtained by sequentially executing the semantic splitting task and the form conversion task based on the same predetermined input text information.

[0094] Obtaining the first prompt information based on the combination of the semantic splitting example and the form conversion example may include: filling the semantic splitting example and the form conversion example into a prompt information template to obtain the first prompt information. In addition, the prompt information template may include description information of the semantic splitting task and the form conversion task.

[0095] For example, still taking the first prompt information A as an example, based on the semantic analysis example provided in the first prompt information A as step 1, the first prompt information may further include a form conversion example, such as, "Step 2. Based on step 1, the questions and answers are: 1. Question: Is there a lady wearing a black shirt? Answer: Yes. 2. Question: Is there a lady cooking? Answer: Yes. 3. Question: Is there a black dog? Answer: Yes. 4. Question: Is the black dog next to the lady? Answer: Yes."

[0096] The embodiment of the present disclosure obtains the first prompt information by combining the semantic splitting example and the formal conversion example, and can help the first large model better understand and perform the semantic splitting task and the formal conversion task based on the first prompt information, thereby obtaining a more accurate question-answer pair.

[0097] According to an embodiment of the present disclosure, obtaining the first prompt information includes: determining, based on the scene type of the input text information, semantic splitting examples and formal conversion examples that match the scene type; and combining the semantic splitting examples and formal conversion examples that match the scene type to obtain the first prompt information.

[0098] Multiple input text messages can correspond to multiple semantic splitting examples and form conversion examples. For example, input text messages of multiple scene types correspond to multiple semantic splitting examples and form conversion examples. The scene type can be understood as the scene of the input text message. For example, the scene type can include life scenes, shopping scenes, translation scenes, etc.

[0099] Common phrases, descriptive conventions, and grammars may differ across various scenario types. Therefore, when obtaining the first prompt information, the scenario type of the input text information can be determined first, and then semantic splitting examples and formal conversion examples that match the scenario type can be determined. For example, a large classification model can be used to determine the scenario type of the input text information, and then the predetermined semantic splitting examples and formal conversion examples for that scenario type can be obtained to combine and generate the first prompt information.

[0100] A scene type may include multiple semantic splitting examples and multiple form conversion examples. Therefore, after determining the scene type of the input text information, one or more groups of semantic splitting examples and form conversion examples for the same input text information can be randomly determined from the scene type; then, these groups are combined into the first prompt information.

[0101] In an embodiment of the present disclosure, the first prompt information is obtained by combining semantic splitting examples and formal conversion examples that match the scene type of the input text information, which can provide examples that are closer to the scene type of the input text information, thereby helping the first large model to better understand and perform semantic splitting tasks and formal conversion tasks, thereby obtaining more accurate question and answer pairs.

[0102] According to an embodiment of the present disclosure, obtaining the first prompt information includes: determining at least one semantic splitting example and at least one formal conversion example based on the text complexity of the input text information; and combining the at least one semantic splitting example and the at least one formal conversion example to obtain the first prompt information.

[0103] The complexity of a text can be determined by the length of the input text or the number of objects it contains. For example, “a woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her” is more complex than “a woman in a black shirt is cooking on the stove.”

[0104] The higher the text complexity, the more complex the semantic relationship and the more detailed information in the input text information. Therefore, based on the text complexity, the number of semantic splitting examples and formal conversion examples can be increased, one or more semantic splitting examples and one or more formal conversion examples can be determined, and the first prompt information can be obtained by combining one or more semantic splitting examples and one or more formal conversion examples. For example, when the text complexity is in a certain complexity interval, the number of semantic splitting examples and the number of formal conversion examples corresponding to the complexity interval can be determined based on the mapping table of the complexity interval.

[0105] In an embodiment of the present disclosure, a corresponding number of semantic splitting examples and formal conversion examples are determined according to the text complexity of the input text information, which can help the first large model understand more complex input text information and perform semantic splitting tasks and formal conversion tasks, thereby obtaining more accurate question-answer pairs.

[0106] Figure 5 The following schematically shows a scenario diagram of obtaining the first prompt information according to an embodiment of the present disclosure. Figure 5 As shown, embodiment 500 generates the first prompt information based on the scene type and text complexity.

[0107] like Figure 5 As shown, based on input text information 501, the scene type 502 and text complexity 503 of the input text information 501 can be determined. Then, the number of semantic split examples and formal conversion examples is determined based on the text complexity 503. Among the multiple semantic split examples and multiple formal conversion examples under the scene type 502, at least one semantic split example and at least one formal conversion example that are consistent with the above number are determined, such as semantic split example 1 504-11 and formal conversion example 1 504-12...semantic split example p 504-p1 and formal conversion example p 504-p2. The above examples are combined to obtain first prompt information 505.

[0108] According to an embodiment of the present disclosure, for operation S240, based on the comparison result between at least one first answer information and at least one second answer information, at least one target image that matches the image search requirement is determined from at least one first candidate image, including: for each first candidate image, determining the comparison result of the first answer information and the second answer information used to answer the same question information; based on the comparison result, determining the first matching degree between each first candidate image and the input text information; and based on the first matching degree, screening out at least one target image that matches the image search requirement from at least one first candidate image.

[0109] For example, the similarity between the first answer and the second answer to the same question can be calculated, and the comparison result can be determined based on the similarity. For example, the comparison result can include identical or different. If the similarity is greater than a certain similarity threshold, the comparison result is identical; otherwise, the comparison result is different.

[0110] In one embodiment, for each first candidate image, a first degree of match between each first candidate image and the input text information is determined based on the number of question information for which the comparison results are identical. For example, the number may be used directly as the first degree of match. In another embodiment, the similarity between the first answer information and the second answer information for answering the same question information may be used directly as the comparison result, and the average of the comparison results for multiple question information may be used as the first degree of match.

[0111] In another embodiment, since the object serving as the subject in the input text information is usually more important, a higher weight can be assigned to the question information including the subject; then, identical comparison results are regarded as 1, and different comparison results are regarded as 0, and the first matching degree between each first candidate image and the input text information is determined by weighted summation to increase the proportion of the subject in the image search process.

[0112] After determining the first matching degree, at least one target image that matches the image search requirement can be directly screened from the at least one first candidate image based on the first matching degree. For example, the at least one first candidate image can be sorted in descending order based on the first matching degree, and the first predetermined number of first candidate images can be selected as the target image.

[0113] In the embodiment of the present disclosure, since each question information represents the local semantics of the input text information, the first matching degree is determined by comprehensively comparing the results of the number of identical question information. This not only allows screening based on the matching degree of local semantics, but also allows the matching degree of overall semantics to be reflected through quantity. Therefore, using the first matching degree for screening can improve the accuracy of image search.

[0114] Figure 6The following schematically shows a scene diagram of determining a target image according to an embodiment of the present disclosure. Figure 6 As shown, in embodiment 600, after semantic analysis of input text information 601 using the first large model, at least one question-answer pair is generated, and at least one question-answer pair corresponds to at least one question information, such as question information 1 602-1...question information M 602-M; and also corresponds to at least one first answer information, such as first answer information 1 604-11...first answer information M 604-M1.

[0115] At least one first candidate image matching the input text information 601 is: first candidate image 1 603-1 ... first candidate image N 603-N. For first candidate image 1 603-1, a second large model is used to perform a text-graphic analysis on first candidate image 1603-1 and at least one question to obtain at least one second answer information, such as second answer information 1604-12 ... second answer information M 604-M2.

[0116] By combining the comparison results 11 605-11, ..., between the first answer information 1 604-11 and the second answer information 1 604-12, and the comparison result M1 605-M1 between the first answer information M 604-M1 and the second answer information M 604-M2, a first matching degree 1 606-1 for the first candidate image 1 603-1 is obtained. Similarly, a first matching degree N 606-N for the first candidate image N can be obtained based on the above method. Based on the first matching degrees 1 606-1 ...., the first matching degrees N 606-N, at least one target image, such as target image 1 607-1 ...., can be selected from the first candidate images 1 603-1 ...., the first candidate images N 603-N.

[0117] According to an embodiment of the present disclosure, for operation S240, the method also includes: using a third large model to perform graphic analysis on the input text information and at least one first candidate image to generate at least one second matching degree, wherein the second matching degree is used to characterize the degree of matching between the semantics of the first candidate image and the input text information; and based on the first matching degree and the second matching degree, screening out at least one target image that matches the image search requirement from the at least one first candidate image.

[0118] The third large model can be a pre-trained large multimodal model (LMM) for processing visual and linguistic modalities. Similar to the second large model, the third large model is used to perform an image-text analysis task to generate at least one second matching score based on the input text information and at least one first candidate image.

[0119] Unlike the second-largest model, the third-largest model's language modality input is the complete input text. Specifically, the second-largest model is used for local semantic verification, while the third-largest model is used for global semantic verification. Compared to the second-largest model, the third-largest model processes more complex input text, allowing it to utilize LMMs with greater processing power and larger scale.

[0120] Both the first and second matching degrees are for a single first candidate object. When screening target images, the first and second matching degrees can be used to comprehensively evaluate the semantic match between the first candidate image and the image search requirement. For example, the first and second matching degrees can be weighted and summed, and the result of the weighted sum can be used to sort at least one first candidate image in descending order, with the first preset number of first candidate images selected as the target image.

[0121] After local semantic verification by the second-largest model, a relatively accurate local semantic verification result is achieved between the first candidate image and the input text. However, given that large models are susceptible to hallucinations, which can lead to content that is inconsistent with the input text or does not conform to the actual semantics, the large model's output may contain information that does not exist in the first candidate image or the input text. This hallucination can lead to errors in the local semantic verification process, resulting in errors in the final image recall and ranking.

[0122] Therefore, in the embodiments of the present disclosure, a third model is used to perform text-image analysis on the input text and at least one first candidate image, generating at least one second matching degree, and further determining whether the image as a whole meets the image search requirements from a global semantic perspective. Furthermore, the at least one target image selected from the at least one first candidate image based on the first and second matching degrees ensures the accuracy of the image search from both global and local semantic perspectives.

[0123] According to an embodiment of the present disclosure, operation S240 also includes: screening out at least one second candidate image from at least one first candidate image based on the first matching degree; performing graphic and text analysis on the input text information and at least one second candidate image using a third large model to generate at least one third matching degree, wherein the third matching degree is used to characterize the degree of matching between the semantics of the second candidate image and the input text information; and screening out at least one target image that matches the image search requirement from at least one second candidate image based on the third matching degree.

[0124] After obtaining a first match degree through local semantic verification, at least one second candidate image can be selected from the at least one first candidate image based on the first match degree. A third model is then used to perform text-image analysis on the input text and the at least one second candidate image to generate at least one third match degree. Similar to the above, the third model performs text-image analysis to generate a third match degree.

[0125] Since the third matching degree is used to characterize the degree of semantic matching between the second candidate image and the input text information, and the second candidate image is an image that has been screened through local semantic verification, at least one target image can be directly screened out from at least one second candidate image based on the third matching degree from a global semantic perspective.

[0126] For example, at least one second candidate image may be sorted in descending order according to the third matching degree, and the first preset number of second candidate images may be taken as target images.

[0127] In the disclosed embodiments, at least one second candidate image is obtained by first screening based on a first match degree of local semantics. Then, a third-largest model is used to perform graphic analysis on the input text and the at least one second candidate image to generate at least one third match degree. This reduces the amount of data required for subsequent global semantic verification using the third-largest model, improving image search speed. Subsequently, the generated third match degree is used to perform screening based on a global semantic perspective, improving image search accuracy from both a local and global perspective.

[0128] For ease of understanding, two embodiments are taken as examples below to illustrate the process of screening and obtaining at least one target image.

[0129] Figure 7A The following schematically shows a scene diagram of determining a target image using the third largest model according to an embodiment of the present disclosure. Figure 7A As shown, in embodiment 700A, a first large model M1 is used to generate a question-answer pair 1 703-1 based on input text information 701 and first prompt information. Question-answer pair 1 703-1 includes question information 703-11 and first answer information 703-12. For first candidate images 1 702-1...first candidate images N 702-N that match input text information 701, a second large model M2 is used to perform a semantic analysis of local semantics on first candidate image 1 702-1 and question information 703-11 to generate second answer information 704-11. Based on the comparison results between first answer information 703-12 and second answer information 704-11, a first matching degree 705-1 is determined for first candidate image 1 702-1.

[0130] Furthermore, a third model M3 is used to perform global semantic verification (text-image analysis) on the input text information 701 and the first candidate image 1 702-1, generating a second matching degree 707-1 for the first candidate image 1 702-1. Based on the first matching degree 705-1 and the second matching degree 707-1, it is determined whether to select the first candidate image 1 as the target image from the first candidate images 1 702-1…first candidate images N 702-N, thereby obtaining target images 1 706-1….

[0131] Figure 7B Schematically shows a scene diagram of determining a target image using the third largest model according to another embodiment of the present disclosure. Figure 7B As shown, in embodiment 700B, the process of obtaining the first matching degree 705-1 of the first candidate image 1 702-1 is similar to that of embodiment 700B and will not be described in detail here. This process is also a process of verifying local semantics.

[0132] After generating the first matching degree 705-1, it can be determined whether to select the first candidate image 1 from the first candidate image 1 702-1...first candidate image N 702-N as the second candidate image according to the first matching degree 705-1, such as obtaining at least one second image: second candidate image 1 708-1...

[0133] In this embodiment, the third large model M3 can be used to perform global semantic verification on the second candidate image 1 708-1 and the input text information 701 to generate a third matching degree 709 for the second candidate image 1 708-1. Based on the third matching degree 709, it can be determined whether to select the second candidate image 1 708-1 from the second candidate images 1 708-1... to obtain at least one target image, such as the target image 1 706-1...

[0134] According to an embodiment of the present disclosure, the method also includes: obtaining second prompt information, wherein the second prompt information is used to prompt the third largest model to perform a text-image analysis task; using the third largest model to perform a text-image analysis task on the input text information and at least one first candidate image based on the second prompt information, and generate at least one second matching degree; or using the third largest model to perform a text-image analysis task on the input text information and at least one second candidate image based on the second prompt information, and generate at least one third matching degree.

[0135] The third largest model can perform the image-text analysis task to generate the second matching degree / third matching degree. At this time, the third largest model can be prompted to perform the image-text analysis task through the second prompt information.

[0136] In addition, the generated second matching degree / third matching degree format may be prompted in the second prompt information to facilitate screening and sorting.

[0137] For example, the second prompt information may be: "Your task is to evaluate and determine whether the given candidate image reflects the text instruction "[[INSTRUCTION]]". The steps are: 1. Please carefully observe the candidate image provided. 2. Determine whether it meets the description of the text instruction. 3. Please provide your answer in the following format: ANSWER: [Yes / No], where: If the candidate image correctly meets the instruction description, select "Yes". If it does not meet the instruction description, select "No"." In the second prompt information, the placeholder "[[INSTRUCTION]]" is the location for filling in the input text information, and Yes / No can be converted into numerical values ​​1 / 0 respectively as the second matching degree / third matching degree.

[0138] For example, if the input text is as follows: A woman in a black shirt is cooking on the stove, with a black dog sitting next to her. The first candidate image and the second candidate image are: A woman in a black shirt is cooking on the stove, with a black dog and a brown dog sitting next to her. In this case, the third largest model outputs ANSWER: [Yes], and the second match degree and the third match degree are 1.

[0139] According to an embodiment of the present disclosure, the method further includes: sequentially arranging at least one target image into an image search sequence according to a matching degree, and outputting the image search sequence, wherein the matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree.

[0140] After selecting at least one target image from at least one first candidate image, the at least one target image may be output. When outputting, the at least one target image may be sequentially organized into an image search sequence in descending order of matching degree and output.

[0141] For example, in an embodiment in which the target image is directly obtained by screening according to the first matching degree, at least one target image may be arranged in descending order according to the first matching degree.

[0142] For another example, in an embodiment where a target image is obtained by screening based on a first matching degree and a second matching degree, at least one target image may be arranged in descending order based on the first matching degree or the second matching degree. Alternatively, the target image may be arranged in descending order based on the weighted sum of the first matching degree and the second matching degree.

[0143] For another example, in an embodiment where a second candidate image is first screened based on a first matching degree and a target image is then screened from the second candidate images based on a third matching degree, the at least one target image may be arranged in descending order based on the first matching degree or the third matching degree. Alternatively, the target image may be arranged in descending order based on the weighted sum of the first matching degree and the third matching degree.

[0144] In this embodiment, the first matching degree reflects the matching degree of local semantics, and the second matching degree / third matching degree reflects the matching degree of global semantics. The arrangement method suitable for the user can be determined based on the user's historical behavior data to reduce the time the user spends searching for key target images and improve the user experience.

[0145] In another specific embodiment, the third model can not only perform the image-text analysis task to generate the second matching degree / third matching degree, but also output whether the first candidate image / second candidate image matches the input text information, and also give the reason for the match or not.

[0146] According to an embodiment of the present disclosure, the second prompt information is also used to prompt the third largest model to perform an interpretation task; the method also includes: using the third largest model to perform an interpretation task on the input text information and at least one first candidate image based on the second prompt information, and generating at least one first interpretation information, the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, the first sub-interpretation information is used to interpret objects with the same semantics as the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics from the input text information and the first candidate image; or, using the third largest model to perform an interpretation task on the input text information and at least one second candidate image based on the second prompt information, and generating at least one second interpretation information, the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, the third sub-interpretation information is used to interpret objects with the same semantics as the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics from the input text information and the second candidate image.

[0147] The second prompt information prompts the third model to perform the interpretation task through the newly added explanation prompt information. At this time, the information in the second prompt information used to prompt the third model to perform the image and text analysis task can be called image and text analysis prompt information.

[0148] For example, the image and text analysis prompt is the same as the second prompt above. The newly added explanation prompt can be: "After the ANSWER line, briefly explain how the candidate image meets or does not meet the text instructions. Important note: Only analyze based on the text instructions and related images. Ignore elements unrelated to the text instructions. Do not introduce content beyond the text instructions." Among them, "Only analyze based on the text instructions and related images. Ignore elements unrelated to the text instructions. Do not introduce content beyond the text instructions. Always start with the ANSWER line and then provide an explanation on a new line." is used to avoid hallucination output.

[0149] For example, the input text is as follows: A woman in a black shirt is cooking on the stove, with a black dog sitting next to her. The first candidate image / second candidate image is: A woman in a black shirt is cooking on the stove, with a black dog and a brown dog sitting next to her. In this case, the third-largest model outputs ANSWER: [Yes], the second match / third match is 1, and the image includes a brown dog, which is not in the text. Here, "the image includes a brown dog, which is not in the text" is the second sub-explanatory information in the first explanation information, or the fourth sub-explanatory information in the second explanation information. In another example, the third-largest model output may be: [Yes], the second match / third match is 1, the image includes a brown dog, which is not in the text, the image includes a black dog, which is in the text, and "the image includes a black dog, which is in the text" is the first sub-explanatory information in the first explanation information, or the third sub-explanatory information in the second explanation information.

[0150] At this point, it should be noted that for the prompt message "Analyze only based on text instructions and related images. Ignore elements unrelated to the text instructions. Do not introduce content beyond the text instructions", the "brown dog" not included in the input text information was ignored during the image and text analysis, and thus a YES answer was obtained, but it was explained when explaining the task to facilitate user understanding.

[0151] In the embodiments of the present disclosure, during the verification process of local semantics and global semantics, not only matching candidate images are returned, but also first explanation information / second explanation information can be generated to analyze inconsistent objects and details between images and texts, thereby improving the interpretability and transparency of the image search process and avoiding the poor experience caused by users having difficulty understanding the success / failure of the image search task due to obtaining a simple similarity value.

[0152] Figure 8 A schematic diagram of a scenario in which the third model according to an embodiment of the present disclosure performs graphic and text analysis tasks and interpretation tasks is shown schematically.

[0153] like Figure 8 As shown, scenario 800 includes two embodiments. In one embodiment, a third model M3 is used to perform a text-image analysis task T3 and an interpretation task T4 on the input text information 801 and the first candidate image 802, respectively, based on the second prompt information, to obtain a second matching degree 804 and first interpretation information 805, respectively. Since the first interpretation information 805 is an explanation of the difference between the first candidate image 802 and the input text information 801, each first candidate image corresponds to a piece of first interpretation information. Therefore, after screening and obtaining at least one target image based on the first matching degree and the second matching degree 804, the first interpretation information of the target image can be obtained based on the correspondence between the first candidate image and the target image, thereby generating a search image sequence.

[0154] In another embodiment, a third large model M3 is used to perform an image-text analysis task T3 and an interpretation task T4 on the input text information 801 and the second candidate image 803 based on the second prompt information, respectively, to obtain a third matching degree 807 and second interpretation information 806. Similar to the previous embodiment, after selecting at least one target image using the third matching degree 807, the second interpretation information of the target image can be obtained based on the correspondence between the second candidate image and the target image, thereby generating a search image sequence.

[0155] According to an embodiment of the present disclosure, the method further includes: sequentially arranging at least one target image and first interpretation information into an image search sequence according to a degree of matching, and outputting the image search sequence; or, sequentially arranging at least one target image and second interpretation information into an image search sequence according to a degree of matching, and outputting the image search sequence.

[0156] The matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree. The sorting method according to the matching degree is described above and will not be repeated here.

[0157] The image search sequence can be arranged in order of information pairs, such as a target image and its corresponding first interpretation information / second interpretation information forming an information pair. When outputting the image search sequence, the first interpretation information / second interpretation information can be displayed near the target image.

[0158] In an embodiment of the present disclosure, the output image search sequence not only includes at least one target image searched, but also includes first explanation information / second explanation information of the searched target image, which makes it convenient for users to easily understand the success / failure of the image search task and its reasons based on the output image search sequence, and provides a better user experience.

[0159] According to an embodiment of the present disclosure, for operation S210, obtaining at least one first candidate image that matches the input text information includes: determining text encoding features of the input text information; and determining at least one first candidate image based on the similarity between the text encoding features and the image encoding features of each image in the image library.

[0160] For example, a text model can be used to encode input text information in a language modality to obtain text encoding features, such as text vectors; an image model can be used to encode images in a visual modality to obtain image encoding features, such as image vectors.

[0161] Alternatively, a large multimodal model can be used to encode the input text and image separately, obtaining text encoding features and image encoding features for each image in the image library. For example, a Vision-Language Model (VLM) consisting of a visual encoder and a language encoder can be used. The internal language encoder can be used to encode the input text to obtain text encoding features, and the internal visual encoder can be used to encode the image to obtain image encoding features.

[0162] The similarity between the text encoding features and the image encoding features can be represented by a distance in a vector space, such as by using a distance metric such as cosine similarity or Euclidean distance. Subsequently, a predetermined preliminary screening threshold can be used to preliminarily screen the image library to obtain at least one first candidate image that matches the input text information.

[0163] In one embodiment, the image coding features of each image in the image library may be pre-coded and stored. When performing image search, the image coding features of each image may be directly obtained and the similarity between the image coding features and the text coding features of the current input text information may be calculated.

[0164] For example, for an input text information T, the visual language model VLM is used to obtain the text encoding features of the input text information T, such as using the normalized text vector express, , d represents the dimension of the text vector, which is determined by the pre-trained visual language model VLM. The image encoding features of P images in the image library can be composed of the image matrix composed of normalized image vectors express, . Then, the text vector With the image matrix Perform a preliminary similarity calculation to obtain the similarity of each image relative to the input text information. The similarity is calculated using the following formula (1):

[0165] (1)

[0166] in, represents the inner product, Represents the similarity function, and the output result s is a vector of length P. , each element in the vector s represents the similarity between the corresponding image and the input text information. Sort the similarity from high to low to get a list of images that match the input text information. Then, N images can be selected as the first candidate images, denoted as .

[0167] In an embodiment of the present disclosure, at least one first candidate image is pre-screened based on the similarity between the text encoding features of the input text image and the image encoding features of each image in the image library. Thus, while ensuring the accuracy of the image search, the image processing amount of subsequent large models is reduced, resource usage is reduced, and the image search speed is improved.

[0168] To facilitate understanding of the present disclosure, the image search process will be described below using a specific embodiment as an example. Figure 9 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown.

[0169] In embodiment 900, the input text information T may be "a woman in a black shirt is cooking on the stove, with a black dog sitting next to her", and at least one first candidate image matching the input text information T obtained through predetermined screening is .

[0170] The first model Reasoner is used to perform semantic analysis and form conversion tasks based on Prompt1 (the first prompt information) and the input text information T. After performing the first step of semantic analysis, M sub-input text information p is obtained. i , using a collection Representation. For example, the sub-input text information in the form of 4 declarative sentences is as follows: "1. There is a lady wearing a black shirt in the picture. 2. The lady in the picture is cooking. 3. There is a black dog in the picture. 4. The black dog is next to the lady in the picture." After performing the second step of the form conversion task, at least one question-answer pair is obtained: "1. Question: Is there a lady wearing a black shirt? | Yes; 2. Question: Is there a lady cooking? | Yes; 3. Question: Is there a black dog? | Yes; 4. Is there a black dog next to the lady? | Yes;". The generated M question-answer pairs can be a collection of question information and the collection of correct first answer information .

[0171] The execution process of the first model can be expressed by the following formula (2):

[0172] (2)

[0173] Using the second largest model to analyze at least one first candidate image Perform graphic and text analysis on at least one question information generated by the first large model to generate second answer information for each question information. Afterwards, compare the first answer information with the second answer information under each question information. For example, for image 1 in the first candidate image, the comparison results of questions 1 to 4 are: different, same, same, different, and the number of question information with the same comparison results is 2; for image 2 in the first candidate image, the comparison results of questions 1 to 4 are: different, different, same, same, and the number of question information with the same comparison results is 2...For image N, the comparison results of questions 1 to 4 are: different, different, same, different, and the number of question information with the same comparison results is 1. In this embodiment, the comparison results of different and same are respectively expressed as " "and" "express.

[0174] The process of determining the second candidate image from the first candidate image can be expressed by the following formulas (3) to (5):

[0175] (3)

[0176] (4)

[0177] (5)

[0178] Among them, the second largest model Verifier is for the first candidate image and problem information Perform graphic analysis to obtain the second answer information , through the indicator function Determine the second answer information for the same question information j and the first answer information Are they the same? If they are the same, it is 1, otherwise it is 0. For each candidate image I, the number of the same first answer information and the same second answer information under M question information is obtained. . The vector representing the first matching degree, . Indicates based on Sort the values ​​in descending order, It means that after local semantic verification, at least one second candidate image is obtained .

[0179] Use the third model Evaluator to evaluate at least one second candidate image based on Prompt2 (second prompt information) Perform image-text analysis on the input text information T to generate a third matching degree for each second candidate image to evaluate the degree of matching between each second candidate image and the input text information in terms of global semantics. Figure 9 As shown, the second candidate image The third matching degree is 0, 1...1, and the figure shows Next to the " "and" "Sequence representation.

[0180] like Figure 9 As shown, using the third matching degree, the order of image 2 can be adjusted forward to obtain at least one target image , and feeds back the image search sequence as output to the user.

[0181] The process of determining the target image from the second candidate image can be expressed by the following formulas (6) to (8):

[0182] (6)

[0183] (7)

[0184] (8)

[0185] in, Indicates the third largest model Evaluator for the second candidate image The third matching degree is obtained after global semantic verification with the input text information T, and its value is 1 (indicating compliance) or 0 (indicating non-compliance). A vector representing the third matching degree of all second candidate images, consisting of 0 or 1. represents the result of re-ranking the second candidate image after obtaining the third matching result through global semantic verification. The meaning is the same as the explanation part of formulas (3) to (5) above.

[0186] In addition, the third model Evaluator can also generate explanation information based on Prompt2 and output the explanation information.

[0187] The embodiments of the present disclosure perform multi-level semantic verification on images at both the local and global semantic levels. Local semantic verification ensures that the searched images meet the image search requirements in terms of specific details, such as object color, shape, and background; global semantic verification confirms the overall semantic relationship, ensuring that the searched target image is not only similar to the input text information in terms of features, but also highly consistent with complex semantic relationships (such as spatial relationships between objects). This effectively improves the matching accuracy of complex semantic relationships and details, making the returned target image more compatible with the image search requirements and improving the image search accuracy. In addition, the output image search sequence includes the target image and explanatory information, making it easier for users to understand the success / failure of the image search task and the reasons for it, providing a better user experience.

[0188] Figure 10 The block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown.

[0189] like Figure 10 As shown, the image search device 1000 includes: an acquisition module 1010 , a semantic analysis module 1020 , a graphic and text analysis module 1030 , and a determination module 1040 .

[0190] The acquisition module 1010 is configured to acquire at least one first candidate image that is similar to input text information input by a user, wherein the input text information is used to represent the image search requirement of the user.

[0191] The semantic analysis module 1020 is used to perform semantic analysis on the input text information using the first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information.

[0192] The image and text analysis module 1030 is used to perform image and text analysis on at least one question information and at least one first candidate image using the second large model to generate second answer information for answering each question information.

[0193] The determination module 1040 is configured to determine at least one target image matching the image search requirement from the at least one first candidate image based on a comparison result between the at least one first answer information and the at least one second answer information.

[0194] According to an embodiment of the present disclosure, the semantic analysis module 1020 includes: a first acquisition submodule, a semantic analysis submodule, and a form conversion submodule.

[0195] The first acquisition submodule is used to acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task.

[0196] The semantic analysis submodule is used to perform a semantic analysis task based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information.

[0197] The form conversion submodule is used to use the first large model to perform a form conversion task on each sub-input text information based on the first prompt information to convert the sub-input text information into a question-answer pair.

[0198] According to an embodiment of the present disclosure, the semantic analysis submodule includes:

[0199] The first semantic analysis unit is used to perform a semantic analysis task based on the first prompt information and the input text information using the first large model, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; wherein the semantic dimensions include at least one of the following: color, shape, and background.

[0200] According to an embodiment of the present disclosure, the acquisition submodule includes: a first combining unit, configured to combine the semantic splitting example and the form conversion example to obtain the first prompt information.

[0201] According to an embodiment of the present disclosure, the acquisition submodule includes: a first example determination unit and a second combination unit.

[0202] The first example determination unit is configured to determine, according to the scene type of the input text information, a semantic split example and a form conversion example that match the scene type.

[0203] The second combining unit is configured to combine the semantic splitting examples and the form conversion examples that match the scene type to obtain the first prompt information.

[0204] According to an embodiment of the present disclosure, the acquisition submodule includes: a second example determination unit and a third combination unit.

[0205] A second example determination unit is configured to determine at least one semantic split example and at least one form conversion example according to the text complexity of the input text information; and

[0206] The third combining unit is configured to combine at least one semantic splitting example and at least one form conversion example to obtain first prompt information.

[0207] According to an embodiment of the present disclosure, the determination module 1040 includes a first determination submodule, a second determination submodule, and a first screening submodule.

[0208] The first determination submodule is configured to determine, for each first candidate image, a comparison result of first answer information and second answer information for answering the same question information, wherein the comparison result includes sameness or difference.

[0209] The second determining submodule is configured to determine a first matching degree of each first candidate image according to the number of question information with the same comparison results.

[0210] The first screening submodule is configured to screen out at least one target image matching the image search requirement from at least one first candidate image according to a first matching degree.

[0211] According to an embodiment of the present disclosure, the determination module 1040 further includes: a first image and text analysis submodule and a second screening submodule.

[0212] The first image-text analysis submodule is used to perform image-text analysis on the input text information and at least one first candidate image using the third model to generate at least one second matching degree, wherein the second matching degree is used to indicate whether the semantics of the first candidate image and the input text information match.

[0213] The second screening submodule is configured to screen out at least one target image matching the image search requirement from at least one first candidate image according to the first matching degree and the second matching degree.

[0214] According to an embodiment of the present disclosure, the determination module 1040 further includes: a third screening submodule, a second image and text analysis submodule, and a fourth screening submodule.

[0215] The third screening submodule is configured to screen out at least one second candidate image from the at least one first candidate image according to the first matching degree.

[0216] The second image-text analysis submodule is used to perform image-text analysis on the input text information and at least one second candidate image using a third large model to generate at least one third matching degree, wherein the third matching degree is used to characterize whether the semantics of the second candidate image and the input text information match.

[0217] The fourth screening submodule is configured to screen out at least one target image matching the image search requirement from the at least one second candidate image according to the third matching degree.

[0218] According to an embodiment of the present disclosure, the determination module 1040 further includes: a second acquisition submodule and a third image and text analysis submodule.

[0219] The second acquisition submodule is used to obtain second prompt information, wherein the second prompt information is used to prompt the third model to perform the image and text analysis task.

[0220] The third image-text analysis submodule is used to use the third largest model to perform an image-text analysis task on the input text information and at least one first candidate image based on the second prompt information to generate at least one second matching degree; or to use the third largest model to perform an image-text analysis task on the input text information and at least one second candidate image based on the second prompt information to generate at least one third matching degree.

[0221] According to an embodiment of the present disclosure, the image search device 1000 also includes: a first output module, used to sequentially organize at least one target image into an image search sequence according to the matching degree, and return the image search sequence to the user, wherein the matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree.

[0222] According to an embodiment of the present disclosure, the second prompt information is also used to prompt the third largest model to perform the interpretation task; the determination module 1040 also includes: a first interpretation submodule or a second interpretation submodule.

[0223] The first interpretation submodule is used to perform an interpretation task on the input text information and at least one first candidate image based on the second prompt information using the third model, and generate at least one first interpretation information, where the first interpretation information is used to interpret objects with the same and / or different semantics in the input text information and the first candidate image.

[0224] The second interpretation submodule is used to perform an interpretation task on the input text information and at least one second candidate image based on the second prompt information using the third model, and generate at least one second interpretation information, where the second interpretation information is used to interpret objects with the same and / or different semantics in the input text information and the second candidate image.

[0225] According to an embodiment of the present disclosure, the image search apparatus 1000 further includes: a second output module or a third output module.

[0226] The second output module is configured to sequentially combine the at least one target image and the first interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

[0227] The third output module is used to sequentially combine the at least one target image and the second interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

[0228] According to an embodiment of the present disclosure, the acquisition module 1010 includes: a feature determination submodule and an image determination submodule.

[0229] The feature determination submodule is used to determine the text coding feature of the input text information. The image determination submodule is used to determine at least one first candidate image based on the similarity between the text coding feature and the image coding feature of each image in the image library.

[0230] Figure 11 The block diagram of the structure of the artificial intelligence agent according to the embodiment of the present disclosure is schematically shown. In the embodiment of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Figure 11 As shown, the AI ​​agent 1100 may include five core modules: an input module 1110 , a control module 1120 , a storage module 1130 , a calculation module 1140 and an output module 1150 .

[0231] Input module 1110 is responsible for receiving or perceiving information such as queries, requests, instructions, signals, or data from the outside world (e.g., users or the external environment) and converting it into a format that AI agent 1100 can understand and process. Input module 1110 is the primary link for AI agent 1100 to interact with the outside world. It enables AI agent 1100 to efficiently and accurately obtain necessary "sensory" information from the outside world and respond to this information.

[0232] In an example, the input module 1110 may input the input text information described above.

[0233] In this example, the control module 1120 is the core support for the AI ​​agent 1100 to handle complex tasks. The control module 1120 can execute the image search method described above.

[0234] In the example, the control module 1120 will continuously interact with the storage module 1130, the computing module 1140, and / or the output module 1150 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 1120 acts as a single initiator to initiate communication with the storage module 1130, the computing module 1140, and / or the output module 1150, and there is no communication coupling between the storage module 1130, the computing module 1140, and the output module 1150.

[0235] In this example, the performance of control module 1120 can be closely related to the large model underlying AI agent 1100. To fully leverage the capabilities of the large model, the internal structure of control module 1120 can be designed to be highly configurable and extensible to handle a variety of tasks and requirements in real-world scenarios, such as the aforementioned semantic analysis tasks, form conversion tasks, image and text analysis tasks, and interpretation tasks.

[0236] The storage module 1130 may be responsible for memorizing information such as historical conversations, event streams, etc. The aforementioned prompt information, multiple seed input text information, question-answer pairs, etc. may be included in the storage module 1130 .

[0237] The operation module 1140 can be regarded as a predefined tool library. As described above, the controls for text encoding and image encoding can be included in the operation module 1140.

[0238] In an example, the output module 1150 may output at least one target image described above.

[0239] The AI ​​agent 1100 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and enhance flexibility and versatility.

[0240] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0241] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0242] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0243] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0244] Figure 12 A block diagram of an electronic device 1200 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0245] like Figure 12As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. RAM 1203 may also store various programs and data required for the operation of device 1200. Computing unit 1201, ROM 1202, and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to bus 1204.

[0246] Various components in device 1200 are connected to an input / output (I / O) interface 1205, including an input unit 1206, such as a keyboard and mouse; an output unit 1207, such as various types of displays and speakers; a storage unit 1208, such as a magnetic disk and optical disk; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0247] Computing unit 1201 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 1201 performs the various methods and processes described above, such as the image search method. For example, in some embodiments, the image search method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by computing unit 1201, one or more steps of the image search method described above may be performed. Alternatively, in other embodiments, computing unit 1201 may be configured to perform the image search method via any other suitable means (e.g., via firmware).

[0248] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0249] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0250] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0251] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0252] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0253] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0254] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0255] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. An image search method, comprising: Acquire at least one first candidate image that matches input text information, wherein the input text information is used to represent an image search requirement; Performing semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; Performing graphic and text analysis on at least one of the question information and at least one of the first candidate images using a second large model to generate second answer information for answering each of the question information; and At least one target image matching the image search requirement is determined from at least one first candidate image according to a comparison result between at least one first answer information and at least one second answer information.

2. The method according to claim 1, wherein The using the first large model to perform semantic analysis on the input text information to generate at least one question-answer pair includes: Acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; performing the semantic analysis task based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information; and The first large model is used to perform the form conversion task on each of the sub-input text information based on the first prompt information to convert the sub-input text information into the question-answer pair.

3. The method according to claim 2, wherein: The performing the semantic analysis task based on the first prompt information and the input text information by using the first large model to extract at least one sub-input text information from the input text information includes: performing the semantic analysis task based on the first prompt information and the input text information using the first large model, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; The semantic dimension includes at least one of the following: color, shape, and background.

4. The method according to claim 2 or 3, wherein: The obtaining of the first prompt information includes: The first prompt information is obtained by combining the semantic splitting example and the form conversion example.

5. The method according to claim 2, wherein: The obtaining of the first prompt information includes: Determining, according to the scene type of the input text information, semantic splitting examples and form conversion examples that match the scene type; and The first prompt information is obtained by combining the semantic splitting examples and the form conversion examples that match the scene type.

6. The method according to claim 2, wherein: The obtaining of the first prompt information includes: Determining at least one semantic splitting example and at least one form conversion example according to the text complexity of the input text information; and The first prompt information is obtained by combining at least one of the semantic splitting examples and at least one of the form conversion examples.

7. The method according to claim 1, wherein The determining, based on a comparison result between at least one first answer information and at least one second answer information, at least one target image matching the image search requirement from at least one first candidate image includes: For each of the first candidate images, determining a comparison result of first answer information and second answer information for answering the same question information; Determining a first matching degree between each of the first candidate images and the input text information according to the comparison result; and At least one target image matching the image search requirement is screened out from at least one of the first candidate images according to the first matching degree.

8. The method according to claim 7, further comprising: Performing a graphic analysis on the input text information and at least one of the first candidate images using a third model to generate at least one second matching degree, wherein the second matching degree is used to represent a degree of semantic matching between the first candidate image and the input text information; and At least one target image matching the image search requirement is screened out from at least one first candidate image according to the first matching degree and the second matching degree.

9. The method according to claim 7, further comprising: screening at least one second candidate image from at least one first candidate image according to the first matching degree; performing a graphic analysis on the input text information and at least one of the second candidate images using a third model to generate at least one third matching degree, wherein the third matching degree is used to represent a degree of semantic matching between the second candidate image and the input text information; as well as At least one target image matching the image search requirement is selected from at least one of the second candidate images according to the third matching degree.

10. The method according to claim 8 or 9, further comprising: Obtaining second prompt information, wherein the second prompt information is used to prompt the third model to perform a graphic and text analysis task; Using a third largest model to perform the image-text analysis task on the input text information and at least one first candidate image based on the second prompt information, and generate at least one second matching degree; or using a third largest model to perform the image-text analysis task on the input text information and at least one second candidate image based on the second prompt information, and generate at least one third matching degree.

11. The method according to claim 7, further comprising: At least one of the target images is sequentially organized into an image search sequence according to a matching degree, and the image search sequence is output, wherein the matching degree includes at least one of the following: the first matching degree, the second matching degree, and the third matching degree.

12. The method according to claim 10, wherein: The second prompt information is further used to prompt the third model to perform an interpretation task; the method further includes: performing the interpretation task on the input text information and at least one of the first candidate images based on the second prompt information using a third model to generate at least one first interpretation information, wherein the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, wherein the first sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the first candidate image; or The third largest model is used to perform the interpretation task on the input text information and at least one second candidate image based on the second prompt information to generate at least one second interpretation information, wherein the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, and the third sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the second candidate image.

13. The method according to claim 12, further comprising: Sequentially composing at least one of the target images and the first interpretation information into an image search sequence according to a matching degree, and outputting the image search sequence; or, At least one of the target images and the second interpretation information are sequentially combined into an image search sequence according to the matching degree, and the image search sequence is output.

14. The method according to any one of claims 1, wherein: The obtaining of at least one first candidate image matching the input text information includes: Determining text encoding features of the input text information; and At least one first candidate image is determined according to the similarity between the text encoding feature and the image encoding feature of each image in the image library.

15. An image search device, comprising: an acquisition module, configured to acquire at least one first candidate image similar to input text information input by a user, wherein the input text information is used to represent the image search requirement of the user; a semantic analysis module, configured to perform semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; a graphic and text analysis module configured to perform graphic and text analysis on at least one of the question information and at least one of the first candidate images using a second large model to generate second answer information for answering each of the question information; and A determination module is configured to determine at least one target image matching the image search requirement from at least one first candidate image based on a comparison result between at least one first answer information and at least one second answer information.

16. The device according to claim 15, wherein The semantic analysis module includes: A first acquisition submodule is configured to acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; a semantic analysis submodule, configured to use the first large model to perform the semantic analysis task based on the first prompt information and the input text information, so as to extract at least one sub-input text information from the input text information; and A form conversion submodule is used to use the first large model to perform the form conversion task on each sub-input text information based on the first prompt information to convert the sub-input text information into the question-answer pair.

17. The device according to claim 16, wherein The semantic analysis submodule includes: a first semantic analysis unit, configured to perform the semantic analysis task based on the first prompt information and the input text information using the first large model, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; The semantic dimension includes at least one of the following: color, shape, and background.

18. The device according to claim 16 or 17, wherein The acquisition submodule includes: The first combining unit is configured to combine the semantic splitting example and the form conversion example to obtain the first prompt information.

19. The device according to claim 16, wherein The acquisition submodule includes: A first example determining unit is configured to determine, according to a scene type of the input text information, a semantic splitting example and a form conversion example that match the scene type; and The second combining unit is configured to combine the semantic splitting examples and the form conversion examples that match the scene type to obtain the first prompt information.

20. The apparatus according to claim 16, wherein The acquisition submodule includes: A second example determining unit is configured to determine at least one semantic splitting example and at least one form conversion example according to the text complexity of the input text information; and The third combining unit is configured to combine at least one of the semantic splitting examples and at least one of the form conversion examples to obtain the first prompt information.

21. The apparatus according to claim 15, wherein The determination module includes: a first determining submodule, configured to determine, for each of the first candidate images, a comparison result between first answer information and second answer information for answering the same question information; a second determining submodule, configured to determine a first matching degree of each first candidate image according to the comparison result; and The first screening submodule is configured to screen out at least one target image that matches the image search requirement from at least one first candidate image according to the first matching degree.

22. The apparatus according to claim 21, wherein the determining module further comprises: a first image-text analysis submodule, configured to perform image-text analysis on the input text information and at least one of the first candidate images using a third model to generate at least one second matching degree, wherein the second matching degree is used to indicate whether the semantics of the first candidate image and the input text information match; and The second screening submodule is configured to screen out at least one target image matching the image search requirement from at least one first candidate image based on the first matching degree and the second matching degree.

23. The apparatus according to claim 21, wherein the determining module further comprises: a third screening submodule, configured to screen out at least one second candidate image from at least one first candidate image according to the first matching degree; a second image-text analysis submodule, configured to perform image-text analysis on the input text information and at least one of the second candidate images using a third model to generate at least one third matching degree, wherein the third matching degree is used to indicate whether the semantics of the second candidate image and the input text information match; as well as The fourth screening submodule is configured to screen out at least one target image that matches the image search requirement from at least one of the second candidate images based on the third matching degree.

24. The apparatus according to claim 22 or 23, wherein the determining module further comprises: A second acquisition submodule is configured to acquire second prompt information, wherein the second prompt information is configured to prompt the third model to perform a graphic analysis task; The third image-text analysis submodule is used to use the third largest model to perform the image-text analysis task on the input text information and at least one first candidate image based on the second prompt information to generate at least one second matching degree; or to use the third largest model to perform the image-text analysis task on the input text information and at least one second candidate image based on the second prompt information to generate at least one third matching degree.

25. The apparatus according to claim 21, wherein the image search apparatus further comprises: The first output module is configured to sequentially organize at least one of the target images into an image search sequence according to a matching degree, and return the image search sequence to the user, wherein the matching degree includes at least one of the following: the first matching degree, the second matching degree, and the third matching degree.

26. The apparatus according to claim 24, wherein The second prompt information is also used to prompt the third model to perform an interpretation task; The determining module further includes: a first interpretation submodule, configured to perform the interpretation task on the input text information and at least one of the first candidate images based on the second prompt information using a third large model, and generate at least one first interpretation information, wherein the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, wherein the first sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the first candidate image; or, The second interpretation submodule is used to perform the interpretation task on the input text information and at least one second candidate image based on the second prompt information using a third large model to generate at least one second interpretation information, wherein the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, and the third sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the second candidate image.

27. The apparatus according to claim 26, wherein the image search apparatus further comprises: a second output module, configured to sequentially combine at least one of the target images and the first interpretation information into an image search sequence according to a matching degree, and return the image search sequence to the user; or, A third output module is configured to sequentially combine at least one of the target images and the second interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

28. The apparatus according to claim 15, wherein The acquisition module includes: a feature determination submodule, configured to determine text encoding features of the input text information; and The image determination submodule is configured to determine at least one of the first candidate images based on a similarity between the text encoding feature and an image encoding feature of each image in the image library.

29. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

30. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

31. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images