Image searching method and device, intelligent agent, electronic equipment and storage medium

Abstract

The invention provides an image search method and device, an intelligent agent, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, image search and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. The method comprises the steps of obtaining at least one first candidate image matched with input text information; performing semantic analysis on the input text information by using the first large model to generate at least one question and answer pair, the question and answer pair comprising question information and first answer information; performing image-text analysis on the at least one piece of question information and the at least one first candidate image by using a second large model to generate second answer information for answering each piece of question information; and according to a comparison result between the at least one piece of first answer information and the at least one piece of second answer information, determining at least one target image matched with the image search requirement from the at least one first candidate image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models, image search and other technical fields, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Specifically, it relates to an image search method, device, intelligent body, electronic device and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, big model technology has also been applied to many fields. For example, image search can be realized with the help of big models.

[0003] However, currently, there are some deviations in details between the images obtained by image search based on large models and the input information, such as semantic inconsistency. Summary of the invention

[0004] The present disclosure provides an image search method, device, intelligent agent, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, there is provided an image search method, comprising: acquiring at least one first candidate image matching input text information, wherein the input text information is used to characterize image search requirements; performing semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; performing graphic-text analysis on at least one question information and at least one first candidate image using a second large model to generate second answer information for answering each question information; and determining at least one target image matching the image search requirements from at least one first candidate image based on a comparison result between at least one first answer information and at least one second answer information.

[0006] According to another aspect of the present disclosure, an image search device is provided, including: an acquisition module for acquiring at least one first candidate image similar to input text information input by a user, wherein the input text information is used to characterize the user's image search needs; a semantic analysis module for performing semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; a graphic-text analysis module for performing graphic-text analysis on at least one question information and at least one first candidate image using a second large model to generate second answer information for answering each question information; and a determination module for determining at least one target image matching the image search needs from at least one first candidate image based on a comparison result between at least one first answer information and at least one second answer information.

[0007] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the method provided by the embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture to which the image search method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 A flowchart of an image search method according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A schematic diagram schematically shows a scene of an image search method according to an embodiment of the present disclosure;

[0016] Figure 4 A schematic diagram of a scenario for generating a question-answer pair according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 5 A schematic diagram of a scenario for obtaining first prompt information according to an embodiment of the present disclosure is schematically shown;

[0018] Figure 6 A schematic diagram of a scene for determining a target image according to an embodiment of the present disclosure is schematically shown;

[0019] Fig. 7A A schematic diagram of a scene for determining a target image using a third largest model according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 7B A schematic diagram of a scene of determining a target image using a third large model according to another embodiment of the present disclosure is schematically shown;

[0021] Figure 8 A schematic diagram schematically shows a scenario in which the third model according to an embodiment of the present disclosure performs a graphic and text analysis task and an interpretation task;

[0022] Fig. 9 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown;

[0023] Fig.10 A block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown;

[0024] Fig.11 A block diagram schematically shows a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0025] Fig.12 A block diagram of an electronic device 1200 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0026] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0027] Although existing image search solutions can search and return images based on text information input by users, these returned images may not meet or fully meet the search intent. For example, the returned images do not meet or fully meet user needs at the semantic level.

[0028] For example, content-based image search matches by extracting low-level features such as color histogram and edge features of the image, while label-based image search uses metadata or manually annotated labels in the image for search. These methods can only process surface features of the image and cannot understand or capture deep semantic features. The above methods are only capable of simpler image matching tasks. For complex language descriptions or combined search tasks, the search accuracy is low. In addition, image search based on visual language models can map data of visual and language modalities to a shared vector space to achieve basic semantic alignment, such as calculating the similarity between visual modality vectors and language modality vectors. However, semantic alignment in vector space alone is relatively rough and cannot ensure precise semantic consistency. For example, subtle image details (such as color, shape, position) or complex semantic relationships (such as interactions between objects) may not be accurately captured and matched.

[0029] In summary, there are semantic deviations in details and complex semantic relationships between the images obtained by existing image search schemes and the input information, and the image search accuracy is low.

[0030] In order to at least partially solve the above technical problems, the embodiment of the present disclosure provides an image search method, including: obtaining at least one first candidate image matching input text information, wherein the input text information is used to characterize the image search requirements; using a first large model to perform semantic analysis on the input text information to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; using a second large model to perform graphic analysis on at least one question information and at least one first candidate image to generate second answer information for answering each question information; and according to the comparison result between at least one first answer information and at least one second answer information, determining at least one target image matching the image search requirements from at least one first candidate image. Therefore, the embodiment of the present disclosure can at least solve the problem of semantic deviation between the output image and the input, and achieve the technical effect of improving the accuracy of image search.

[0031] Figure 1 An exemplary system architecture to which the image search method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0032] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the image search method and apparatus can be applied may include a terminal device, but the terminal device may implement the method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0033] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0034] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0035] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0037] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0038] It should be noted that the image search method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the image search device provided in the embodiment of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0039] Alternatively, the image search method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the image search device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The image search method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image search device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] For example, a user inputs input text information for image search through the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the first terminal device 101, the second terminal device 102, and the third terminal device 103 can obtain at least one first candidate image matching the input text information, wherein the input text information is used to characterize the image search requirements; use the first large model to perform semantic analysis on the input text information to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; use the second large model to perform graphic analysis on at least one question information and at least one first candidate image to generate second answer information for answering each question information; and determine at least one target image matching the image search requirements from at least one first candidate image based on the comparison result between at least one first answer information and at least one second answer information.

[0041] Alternatively, a document is sent to the server 105 through the first terminal device 101, the second terminal device 102, and the third terminal device 103 to obtain input text information for image search, and the above-mentioned image search method is performed with the help of the server 105 to determine and return at least one searched target image to the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0042] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0043] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0044] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0045] Figure 2 The flowchart of the image search method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0047] In operation S210, at least one first candidate image matching input text information is acquired, wherein the input text information is used to represent an image search requirement.

[0048] The input text information includes text information obtained by the user through at least one input operation, for example, text information obtained by combining after clarifying the image search requirements based on at least one round of dialogue. Alternatively, the input text information may also be text information obtained by processing the input multimodal information, such as converting the information of the visual modality and the audio modality into the input text information of the language modality.

[0049] In an application scenario, a user can communicate with a digital human / virtual human through a 2D perspective such as a display interface or a 3D perspective such as virtual augmented reality to input multimodal information, thereby directly obtaining text information in a language modality, or converting information in other modalities into text information in a language modality to obtain input text information.

[0050] Image search needs, also known as image search intent, include a variety of needs related to image search tasks, such as searching for images similar to a certain image, searching for images of a certain category, searching for images of a certain product, etc.

[0051] The first candidate image may be an image that matches the input text information after a pre-screening operation. For example, a method of obtaining at least one first candidate image that matches the input text information may include: screening out images of the same type, size, and source as the input text information from multiple images to obtain the first candidate image. Alternatively, it may be an image obtained after a rough screening at the semantic level; for example, semantically aligning the input text information and multiple images in the vector space, and screening out images similar to the input text information, and using the similar images as the first candidate images.

[0052] In operation S220, a semantic analysis is performed on the input text information using the first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information.

[0053] The first large model can be a pre-trained single-modal large model, such as a large language model (LLMs) of a language modality; it can also be a pre-trained multimodal large model (Large Multimodal Models, LMMs) that can process language modalities.

[0054] The first large model can perform semantic analysis on the input text information as the model input, and output at least one question-answer pair. Specifically, the semantic analysis can be information splitting / extraction of the semantic dimension of the input text information, and outputting paired question information and first answer information. Each question-answer pair output by the first large model is used to represent the result obtained after the local semantic analysis in the input text information.

[0055] In an illustrative embodiment, the question information in the question-answer pair contains less information than the input text information, such as the number of objects and the number of attributes of the objects contained are less than those in the input text information.

[0056] For example, the input text information is "Person 1 is eating at the dining table", and the question and answer pairs output by the first model include: "Is Person 1 eating? Yes", "Is the food on the dining table? Yes", "Is Person 1 at the dining table? Yes", and the question information is "Is Person 1 eating?", "Is the food on the dining table?", "Is Person 1 at the dining table?", and the first answer information is "Yes", "Yes", and "Yes" respectively.

[0057] In operation S230, a second large model is used to perform graphic and text analysis on at least one question information and at least one first candidate image to generate second answer information for answering each question information.

[0058] The second largest model is a multimodal large model LMMs capable of processing visual modalities and language modalities, and the second largest model is a pre-trained multimodal large model. The input of the second largest model is multimodal information, such as at least one question information and at least one candidate image.

[0059] The second large model performs a text-image analysis on at least one question information and at least one candidate image through a text-image analysis task. The text-image analysis task may be to understand the information in the first candidate image and answer the question information.

[0060] In an illustrative embodiment, the second large model can perform a graphic analysis task based on a single question information and a single first candidate image to generate second answer information that answers the question information; or the second large model can also generate second answer information that answers each question information based on multiple question information and a single first candidate image after understanding the content of the first candidate image. For example, multiple question information and a single first candidate image can be input into the second large model in an array.

[0061] For example, taking question information 1 as "Is person 1 at the dinner table?", after the second largest model understands the first candidate image 1, it generates the second answer information to answer the above question information 1 as "no"; after understanding the first candidate image 2, it generates the second answer information to answer the above question information 1 as "yes".

[0062] In operation S240, at least one target image matching the image search requirement is determined from the at least one first candidate image according to the comparison result between the at least one first answer information and the at least one second answer information.

[0063] The number of the first answer information and the second answer information is the same, which are two pieces of answer information for the same question information from the first large model and the second large model. Therefore, the first answer information and the second answer information for the same question information can be compared, so as to screen out at least one target image matching the image search requirement from at least one first candidate image.

[0064] For example, for at least one question information, the comparison result between the first answer information and the second answer information under each question information can be integrated to screen out at least one target image. The comparison result can be the similarity between the first answer information and the second answer information. For example, at least one target image can be screened out by integrating the similarity between the first answer information and the second answer information under each question information. For another example, the similarity between the first answer information and the second answer information under each question information is weighted and summed, and the target image is screened according to the result of the summation. Alternatively, according to whether the first answer information and the second answer information under each question are the same, and the number of the same or different ones is counted, at least one target image is screened out. In addition, the number of target images can be determined based on a predetermined threshold or screening number.

[0065] In the embodiments of the present disclosure, since the first large model splits the input text information into at least one question-answer pair, the question information in the question-answer pair has a finer semantic granularity than the input text information, and the first answer information and the second answer information are also responses to the finer-grained question information. Therefore, based on the comparison results of the first answer information and the second answer information, it is possible to perform screening at a finer-grained local semantic perspective, thereby selecting a target image with a smaller local semantic deviation from the first candidate image, and the image search accuracy is higher. In addition, since the comparison results of the first answer information and the second answer information under at least one question information are integrated, the embodiments of the present disclosure can also ensure a target image with a smaller overall semantic deviation, and the image search accuracy is higher.

[0066] In addition, before using the second largest model for image and text analysis, a preliminary screening is performed to obtain the first candidate image that matches the input text information. This ensures the accuracy of the image search while reducing the amount of data processed by the second largest model, making the image search faster.

[0067] According to an embodiment of the present disclosure, for operation S220, using the first large model to perform semantic analysis on the input text information to generate at least one question-answer pair includes: obtaining first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; using the first large model to perform a semantic analysis task based on the first prompt information and the input text information to extract at least one sub-input text information from the input text information; and using the first large model to perform a form conversion task on each sub-input text information based on the first prompt information to convert the sub-input text information into a question-answer pair.

[0068] The first large model is used to sequentially perform the semantic analysis task and the form conversion task. For example, the first large model can be prompted to sequentially perform the above two tasks through the pre-built first prompt information. The first prompt information can be directly obtained when using the first large model.

[0069] The first prompt information reserves a position for filling in the input text information. For example, the first prompt information includes a placeholder [[INSTRUCTION]]. After obtaining the first prompt information, the current input text information can be filled in the placeholder [[INSTRUCTION]] to obtain a new input.

[0070] The semantic analysis task, i.e., the first model, understands and analyzes the semantics of the input text information, and extracts at least one sub-input text information from the input text information according to the semantics. Each sub-input text information includes part of the semantics of the input text information.

[0071] The form of the sub-input text information may be a declarative sentence. The form of the sub-input text information may be specified in the first prompt information. For example, the information in the first prompt information used to prompt the first large model to perform the semantic analysis task may be "decompose the input text instruction into multiple verifiable declarative sentences", where "text instruction" is the input text information, and "declarative sentence" is the form of the sub-input information.

[0072] In an exemplary embodiment, the sub-input text information may be directly decomposed from the input text information, or may be extracted from the input text information and then reassembled.

[0073] The form conversion task is used to convert the sub-input text information obtained by the semantic analysis task into a predetermined form, such as a question-answer pair.

[0074] In an exemplary implementation, the first prompt information may be used to limit the form of the first answer information so that it is the same as or similar to the form of the second answer information, so as to facilitate subsequent comparison of the first answer information and the second answer information.

[0075] For ease of understanding, an example of the first prompt information is given below. For example, the first prompt information may be: "#Task description: The semantic analysis task involves decomposing a text instruction into multiple simple and verifiable propositions, each of which has a unique answer of "yes" or "no". According to the instructions provided, you need to decompose the instruction into several atomic propositions and corresponding answers, following the two steps below. ## Step 1: Declarative sentence conversion: Decompose the input text instruction into multiple verifiable declarative sentences. ## Step 2: Question form conversion: Convert each declarative sentence into a question form and provide the correct answer according to the given instructions. Text instructions will be provided to you to complete this task: Enter text information." Step 1 and Step 2 correspond to the semantic analysis task and the form conversion task, respectively.

[0076] In an embodiment of the present disclosure, the first prompt information is used to enable the first large model to sequentially perform semantic analysis tasks and form conversion tasks to obtain at least one question-answer pair. Through this hierarchical sequential task execution method, the first large model can better understand the complex semantic relationships and detail information in the input text information, thereby improving the accuracy of local semantic extraction of question information in the question-answer pair. In addition, converting the sub-input text information into the form of a question-answer pair facilitates subsequent graphic analysis of the question information and comparison of the second answer information with the first answer information.

[0077] Figure 3 The following is a schematic diagram of a scene of an image search method according to an embodiment of the present disclosure. Figure 3As shown, the embodiment 300 is used to search for at least one target image that matches the image search requirement represented by the input text information 301, such as target image 1 306-1...

[0078] In this embodiment, the input text information 301 and the first prompt information are input into the first large model M1. The first large model M1 can perform a semantic analysis task T1 according to the first prompt information and the input text information 301 to obtain at least one sub-input text information, such as sub-input text information 1 ... sub-input text information M. Afterwards, the first large model M1 can perform a form conversion task T2 on each sub-input text information, such as the first large model M1 performs a form conversion task T2 on sub-input text information 1 to obtain a question-answer pair 1 303-1, which includes question information 303-11 and first answer information 303-12.

[0079] In addition, for the input text information 301, at least one first candidate image matching the input text information 301 may be screened out, such as first candidate image 1 302-1 . . . first candidate image N 302-N.

[0080] After the first large model M1 outputs the question information 303-11, the question information 303-11 and the first candidate image 1 302-1 can be input into the second large model M2. After understanding the first candidate image 1 302-1, the second large model M2 generates the second answer information 304-11 for answering the question information 303-11, and compares the first answer information 303-12 of the question information 303-11 with the second answer information 304-11 to obtain the comparison result 305-1. Similarly, the process of obtaining the comparison results between other question information and other first candidate images is as above, which will not be repeated here.

[0081] After obtaining the comparison result between at least one first answer information and at least one second answer information, the comparison result can be used to screen out at least one target image from at least one first candidate image. For example, according to the comparison result 305-1..., it is determined whether to select the first candidate image 301-1 as the target image from the first candidate image 1 302-1...the first candidate image N 302-N.

[0082] According to an embodiment of the present disclosure, a semantic analysis task is performed based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information, including: performing a semantic analysis task based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; wherein the semantic dimension includes at least one of the following: color, shape, background.

[0083] The input text information may include at least one object, such as a person, an animal, a plant, an object, etc. In some embodiments, the first large model may extract at least one sub-input text information from the input text information based on the perspective of multiple objects, such as understanding the complex semantic relationship between objects. In order to better understand the local semantics from multiple semantic dimensions, the limitation of the semantic dimension may be added to the first prompt information, so that the first large model can extract at least one sub-input text information from multiple semantic dimensions.

[0084] For example, the first prompt information may include a prompt of a semantic dimension, such as the first prompt information may include "decompose the input text instruction into multiple verifiable declarative sentences, and the multiple declarative sentences cover at least one of the following semantic dimensions: color, shape, background."

[0085] Alternatively, the first prompt information may also provide a semantic analysis example to help the first large model understand how to extract at least one sub-input text information from the input text information based on multiple semantic dimensions.

[0086] For example, the first prompt information A may be "decompose the input text instruction into multiple verifiable declarative sentences. The following is an example for reference: For the text instruction - a woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her. The decomposed declarative sentences are: There is a lady in a black shirt in the picture. The lady in the picture is cooking. There is a black dog in the picture. The black dog in the picture is next to the lady. Please query according to ##: Execute text instruction: [[INSTRUCTION]]" It can be understood that the sub-text input information in the example includes the objects "woman" and "dog", the colors include "black shirt" and "black dog", and the background includes "black dog next to the lady".

[0087] In the embodiments of the present disclosure, since at least one sub-input text information is based on multiple semantic dimensions, the semantic dimensions of the question information obtained after converting the sub-input text information are richer and can focus on complex semantic relationships and detail information, thereby improving the accuracy of the question information and helping to improve the accuracy of subsequent image searches.

[0088] Figure 4 The following schematically shows a scenario diagram of generating a question-answer pair according to an embodiment of the present disclosure. Figure 4As shown, in the embodiment 400, the first large model M1 performs the semantic analysis task T1 according to the input text information 401 and the first prompt information, and obtains multiple sub-input text information for multiple objects. For example, object 1 402-1 includes sub-input text information 1 402-11 under semantic dimension 1...sub-input text information a 402-1 a under semantic dimension a; object 2 402-2 includes sub-input text information...

[0089] For each semantic dimension, the first large model M1 can be used to perform the form conversion task T2 to output a question-answer pair. For example, for the sub-input text information 1 402-11, after performing the form conversion task T2, a question-answer pair 1 403-1 is obtained, wherein the question-answer pair 1 403-1 includes question information 403-11 and first answer information 403-12.

[0090] According to an embodiment of the present disclosure, obtaining the first prompt information includes: combining a semantic splitting example and a form conversion example to obtain the first prompt information.

[0091] Examples of semantic splitting include: at least one sub-input text information obtained after performing a semantic splitting task based on predetermined input text information, or at least one sub-input text information of multiple semantic dimensions.

[0092] The form conversion example includes: a question-answer pair obtained after performing a form conversion task based on predetermined sub-input text information.

[0093] The semantic splitting example and the form conversion example are the results obtained by sequentially executing the semantic splitting task and the form conversion task based on the same predetermined input text information.

[0094] Obtaining the first prompt information according to the combination of the semantic splitting example and the form conversion example may include: filling the semantic splitting example and the form conversion example into a prompt information template to obtain the first prompt information. In addition, the prompt information template may include description information of the semantic splitting task and the form conversion task.

[0095] For example, still taking the first prompt information A as an example, based on the semantic analysis example given in the first prompt information A, as step 1, the first prompt information may also include a form conversion example such as, "Step 2. Based on step 1, the questions and answers are: 1. Question: Is there a lady wearing a black shirt? Answer: Yes. 2. Question: Is there a lady cooking? Answer: Yes. 3. Question: Is there a black dog? Answer: Yes. 4. Question: Is the black dog next to the lady? Answer: Yes."

[0096] The embodiments of the present disclosure obtain the first prompt information by combining the semantic splitting examples and the formal conversion examples, and can help the first large model to better understand and perform the semantic splitting task and the formal conversion task based on the first prompt information, thereby obtaining more accurate question-answer pairs.

[0097] According to an embodiment of the present disclosure, obtaining the first prompt information includes: determining, according to the scene type of the input text information, semantic splitting examples and formal conversion examples that match the scene type; and combining the semantic splitting examples and formal conversion examples that match the scene type to obtain the first prompt information.

[0098] Multiple input text information may correspond to multiple semantic splitting examples and form conversion examples. For example, input text information of multiple scene types may correspond to multiple semantic splitting examples and form conversion examples. The scene type may be understood as the scene of the input text information, for example, the scene type may include life scenes, shopping scenes, translation scenes, etc.

[0099] There may be differences in common sentences, description habits, grammar, etc. for various scene types. Therefore, when obtaining the first prompt information, the scene type of the input text information can be determined first, and then the semantic splitting examples and form conversion examples that match the scene type can be determined. For example, the scene type of the input text information can be determined using a classification model, and then the semantic splitting examples and form conversion examples predetermined under the scene type can be obtained to combine and obtain the first prompt information.

[0100] A scene type may include multiple semantic splitting examples and multiple form conversion examples. Therefore, after determining the scene type of the input text information, one or more groups of semantic splitting examples and form conversion examples for the same input text information may be randomly determined from the scene type; and then, they are combined into the first prompt information.

[0101] In an embodiment of the present disclosure, the first prompt information is obtained by combining semantic splitting examples and formal conversion examples that match the scene type of the input text information, which can provide examples that are closer to the scene type of the input text information, thereby helping the first large model to better understand and perform semantic splitting tasks and formal conversion tasks, thereby obtaining more accurate question and answer pairs.

[0102] According to an embodiment of the present disclosure, obtaining the first prompt information includes: determining at least one semantic splitting example and at least one formal conversion example according to the text complexity of the input text information; and combining the at least one semantic splitting example and the at least one formal conversion example to obtain the first prompt information.

[0103] The complexity of the text can be determined by the length of the input text information or the number of objects contained in the input text information. For example, "a woman in a black shirt is cooking on the stove and there is a black dog sitting next to her" is more complex than "a woman in a black shirt is cooking on the stove".

[0104] The higher the text complexity, the more complex the semantic relationship and the more detailed information in the input text information. Therefore, the number of semantic splitting examples and formal conversion examples can be increased based on the text complexity, one or more semantic splitting examples, one or more formal conversion examples can be determined, and the first prompt information can be obtained by combining one or more semantic splitting examples and one or more formal conversion examples. For example, when the text complexity is in a certain complexity interval, the number of semantic splitting examples and the number of formal conversion examples corresponding to the complexity interval can be determined according to the mapping table of the complexity interval.

[0105] In an embodiment of the present disclosure, a corresponding number of semantic splitting examples and formal conversion examples are determined according to the text complexity of the input text information, which can help the first large model understand more complex input text information and perform semantic splitting tasks and formal conversion tasks, thereby obtaining more accurate question-answer pairs.

[0106] Figure 5 The following schematically shows a scenario diagram of obtaining the first prompt information according to an embodiment of the present disclosure. Figure 5 As shown, embodiment 500 generates the first prompt information according to the scene type and the text complexity.

[0107] like Figure 5 As shown, according to the input text information 501, the scene type 502 and the text complexity 503 of the input text information 501 can be determined. Then, the number of semantic split examples and formal conversion examples is determined by the text complexity 503, and at least one semantic split example and at least one formal conversion example consistent with the above number are determined from the multiple semantic split examples and multiple formal conversion examples under the scene type 502, such as semantic split example 1 504-11 and formal conversion example 1 504-12...semantic split example p 504-p1 and formal conversion example p 504-p2, and the above examples are combined to obtain the first prompt information 505.

[0108] According to an embodiment of the present disclosure, for operation S240, based on the comparison result between at least one first answer information and at least one second answer information, at least one target image matching the image search requirement is determined from at least one first candidate image, including: for each first candidate image, determining the comparison result between the first answer information and the second answer information used to answer the same question information; based on the comparison result, determining a first matching degree between each first candidate image and the input text information; and based on the first matching degree, screening out at least one target image matching the image search requirement from at least one first candidate image.

[0109] For example, the similarity between the first answer information and the second answer information that answer the same question information can be calculated, and the comparison result can be determined based on the similarity. For example, the comparison result includes the same or different. If the similarity is greater than a certain similarity threshold, the comparison result is the same; otherwise, the comparison result is different.

[0110] In one embodiment, for each first candidate image, the first matching degree between each first candidate image and the input text information is determined according to the number of question information with the same comparison result. For example, the number is directly used as the first matching degree. In another embodiment, the similarity between the first answer information and the second answer information for answering the same question information can be directly used as the comparison result, and the average value of the comparison results of multiple question information is used as the first matching degree.

[0111] In another embodiment, since the object serving as the subject in the input text information is usually more important, a higher weight can be given to the question information including the subject; then, the same comparison results are regarded as 1, and different comparison results are regarded as 0, and the first matching degree between each first candidate image and the input text information is determined by weighted summation to increase the proportion of the subject in the image search process.

[0112] After determining the first matching degree, at least one target image matching the image search requirement can be directly screened out from at least one first candidate image according to the first matching degree. For example, at least one first candidate image is sorted in descending order according to the first matching degree, and the first preset number of first candidate images are taken as target images.

[0113] In the embodiments of the present disclosure, since each question information represents the local semantics of the input text information, the first degree of match is determined by comprehensively comparing the results of the same question information. This not only allows screening based on the degree of match of the local semantics, but also allows the degree of match of the overall semantics to be reflected through quantity. Thus, screening using the first degree of match can improve the accuracy of image search.

[0114] Figure 6The following schematically shows a scene diagram of determining a target image according to an embodiment of the present disclosure. Figure 6 As shown, in embodiment 600, after semantic analysis of input text information 601 using the first large model, at least one question-answer pair is generated, and at least one question-answer pair corresponds to at least one question information, such as question information 1 602-1...question information M 602-M; and also corresponds to at least one first answer information, such as first answer information 1 604-11...first answer information M 604-M1.

[0115] At least one first candidate image matching the input text information 601 is: first candidate image 1 603-1 ... first candidate image N 603-N. For the first candidate image 1 603-1, at least one second answer information is obtained by performing a graphic analysis on the first candidate image 1603-1 and at least one question information using the second large model, such as second answer information 1604-12 ... second answer information M 604-M2.

[0116] By combining the comparison results 11 605-11, ..., between the first answer information 1 604-11 and the second answer information 1 604-12, and the comparison results M1 605-M1 between the first answer information M 604-M1 and the second answer information M 604-M2, a first matching degree 1 606-1 of the first candidate image 1 603-1 is obtained. Similarly, a first matching degree N 606-N of the first candidate image N can be obtained based on the above method. According to the first matching degree 1 606-1 ... the first matching degree N 606-N, at least one target image, such as target image 1 607-1 ..., can be screened out from the first candidate image 1 603-1 ... the first candidate image N 603-N.

[0117] According to an embodiment of the present disclosure, for operation S240, the method also includes: using a third large model to perform graphic analysis on the input text information and at least one first candidate image to generate at least one second matching degree, wherein the second matching degree is used to characterize the degree of matching between the semantics of the first candidate image and the input text information; and based on the first matching degree and the second matching degree, screening out at least one target image that matches the image search requirement from the at least one first candidate image.

[0118] The third largest model may be a large multimodal model (Large Multimodal Models, LMMs) for processing visual modalities and language modalities, and the third largest model is a pre-trained large multimodal model. Similar to the second largest model, the third largest model is used to perform a text-image analysis task to generate at least one second matching degree based on the input text information and at least one first candidate image.

[0119] Unlike the second largest model, the third largest model’s language modality input is the complete input text information, that is, the second largest model is used for local semantic verification, and the third largest model is used for global semantic verification. Compared with the second largest model, the third largest model processes more complex input text information, so the third largest model can adopt LMMs with stronger processing capabilities and larger scale.

[0120] The first matching degree and the second matching degree are both for a single first candidate object. When screening the target image, the semantic matching degree between the first candidate image and the image search requirement can be comprehensively evaluated based on the first matching degree and the second matching degree. For example, the first matching degree and the second matching degree can be weighted and summed, and at least one first candidate image can be sorted in descending order based on the result of the weighted summation, and the first preset number of first candidate images can be taken as the target image.

[0121] After the local semantic verification of the second largest model, there is a more accurate local semantic verification result between the first candidate image and the input text information. However, considering that the large model is easily affected by hallucinations, it may produce content that is inconsistent with the input text information or does not conform to the actual semantics, that is, the information output by the large model may not exist in the first candidate image or the input text information. The hallucination output phenomenon may cause errors in the local semantic verification process, which in turn causes errors in the final image recall and ranking.

[0122] Therefore, in the embodiment of the present disclosure, the third model is used to perform text-image analysis on the input text information and at least one first candidate image, and at least one second matching degree is generated, and whether the image meets the image search requirement as a whole is further determined from the global semantic perspective. Furthermore, at least one target image selected from at least one first candidate image according to the first matching degree and the second matching degree can ensure the accuracy of the image search from both the global semantics and the local semantics.

[0123] According to an embodiment of the present disclosure, operation S240 also includes: based on the first matching degree, selecting at least one second candidate image from at least one first candidate image; using a third large model to perform graphic analysis on the input text information and at least one second candidate image to generate at least one third matching degree, wherein the third matching degree is used to characterize the degree of matching between the semantics of the second candidate image and the input text information; and based on the third matching degree, selecting at least one target image that matches the image search requirement from at least one second candidate image.

[0124] After obtaining the first matching degree through local semantic verification, at least one second candidate image can be screened out from at least one first candidate image based on the first matching degree. Afterwards, the third largest model is used to perform text-image analysis on the input text information and at least one second candidate image to generate at least one third matching degree. Similar to the above, the third largest model performs the text-image analysis task to generate the third matching degree.

[0125] Since the third matching degree is used to characterize the degree of matching between the semantics of the second candidate image and the input text information, and the second candidate image is an image that has been screened after local semantic verification, at least one target image can be directly screened out from at least one second candidate image based on the third matching degree from a global semantic perspective.

[0126] For example, at least one second candidate image may be sorted in descending order according to the third matching degree, and the first preset number of second candidate images may be taken as target images.

[0127] In the embodiment of the present disclosure, by first screening based on the first matching degree of local semantics to obtain at least one second candidate image, and then using the third largest model to perform graphic analysis on the input text information and at least one second candidate image to generate at least one third matching degree, the amount of data for subsequent global semantic verification using the third largest model can be reduced, and the image search speed can be improved. Afterwards, the generated third matching degree is used to screen from a global semantic perspective, which can improve the accuracy of image search in both local semantics and global semantics.

[0128] For ease of understanding, two embodiments are taken as examples below to illustrate the process of screening and obtaining at least one target image.

[0129] Fig. 7A The schematic diagram of a scene in which a target image is determined using the third largest model according to an embodiment of the present disclosure is shown schematically. Fig. 7A As shown, in embodiment 700A, the first large model M1 is used to generate a question-answer pair 1 703-1 based on the input text information 701 and the first prompt information, and the question-answer pair 1 703-1 includes question information 703-11 and first answer information 703-12. For the first candidate image 1 702-1 ... first candidate image N 702-N that matches the input text information 701, the second large model M2 is used to perform a semantic analysis of local semantics on the first candidate image 1 702-1 and the question information 703-11 to generate the second answer information 704-11. According to the comparison result between the first answer information 703-12 and the second answer information 704-11, the first matching degree 705-1 of the first candidate image 1 702-1 is determined.

[0130] In addition, the third largest model M3 is used to perform global semantic verification (text-image analysis) on the input text information 701 and the first candidate image 1 702-1, and generate a second matching degree 707-1 of the first candidate image 1 702-1. According to the first matching degree 705-1 and the second matching degree 707-1, it can be determined whether to select the first candidate image 1 as the target image from the first candidate image 1 702-1 ... the first candidate image N 702-N to obtain the target image 1 706-1 ....

[0131] Figure 7B FIG. 2 schematically shows a scene diagram of determining a target image using a third large model according to another embodiment of the present disclosure. Figure 7B As shown, in the embodiment 700B, the process of obtaining the first matching degree 705-1 of the first candidate image 1 702-1 is similar to the embodiment 700B, and will not be repeated here. This process is also a process of verifying the local semantics.

[0132] After the first matching degree 705-1 is generated, it can be determined whether to select the first candidate image 1 from the first candidate image 1 702-1 ... first candidate image N 702-N as the second candidate image according to the first matching degree 705-1, such as obtaining at least one second image: second candidate image 1 708-1 ... .

[0133] In this embodiment, the third large model M3 can be used to perform global semantic verification on the second candidate image 1 708-1 and the input text information 701 to generate a third matching degree 709 of the second candidate image 1 708-1. According to the third matching degree 709, it can be determined whether to select the second candidate image 1 708-1 from the second candidate images 1 708-1... to obtain at least one target image, such as the target image 1 706-1...

[0134] According to an embodiment of the present disclosure, the method also includes: obtaining second prompt information, wherein the second prompt information is used to prompt the third largest model to perform a text-image analysis task; using the third largest model to perform a text-image analysis task on the input text information and at least one first candidate image based on the second prompt information, and generating at least one second matching degree; or, using the third largest model to perform a text-image analysis task on the input text information and at least one second candidate image based on the second prompt information, and generating at least one third matching degree.

[0135] The third largest model can perform the image and text analysis task to generate the second matching degree / third matching degree. At this time, the third largest model can be prompted to perform the image and text analysis task through the second prompt information.

[0136] In addition, the generated second matching degree / third matching degree format may be prompted in the second prompt information to facilitate screening and sorting.

[0137] For example, the second prompt information may be: "Your task is to evaluate and determine whether the given candidate image reflects the text instruction "[[INSTRUCTION]]". The steps are: 1. Please carefully observe the candidate image provided. 2. Determine whether it meets the description of the text instruction. 3. Please provide your answer in the following format: ANSWER: [Yes / No], where: If the candidate image correctly meets the instruction description, select "Yes". If it does not meet the instruction description, select "No"." In the second prompt information, the placeholder "[[INSTRUCTION]]" is the location for filling in the input text information, and Yes / No can be converted into numerical values ​​1 / 0 respectively as the second matching degree / third matching degree.

[0138] For example, the input text information is as follows: A woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her. The first candidate image / second candidate image is: A woman in a black shirt is cooking on the stove, and there is a black dog and a brown dog sitting next to her. At this time, the third largest model outputs ANSWER: [Yes], and the second matching degree / third matching degree is 1.

[0139] According to an embodiment of the present disclosure, the method further includes: sequentially arranging at least one target image into an image search sequence according to a matching degree, and outputting the image search sequence, wherein the matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree.

[0140] After selecting at least one target image from at least one first candidate image, the at least one target image can be outputted. When outputting, the at least one target image can be sequentially organized into an image search sequence in descending order of matching degree and outputted.

[0141] For example, in an embodiment in which the target image is directly screened according to the first matching degree, at least one target image may be arranged in descending order according to the first matching degree.

[0142] For another example, in an embodiment where a target image is obtained by screening according to the first matching degree and the second matching degree, at least one target image may be arranged in descending order according to the first matching degree or the second matching degree. Alternatively, the target image may be arranged in descending order according to the result of the weighted sum of the first matching degree and the second matching degree.

[0143] For another example, in an embodiment where the second candidate image is first screened based on the first matching degree, and then the target image is screened from the second candidate image based on the third matching degree, at least one target image may be arranged in descending order according to the first matching degree or the third matching degree. Alternatively, the target image may be arranged in descending order according to the result of the weighted sum of the first matching degree and the third matching degree.

[0144] In this embodiment, the first matching degree reflects the matching degree of local semantics, and the second matching degree / third matching degree reflects the matching degree of global semantics. The arrangement method suitable for the user can be determined based on the user's historical behavior data to reduce the time the user spends searching for key target images and improve the user experience.

[0145] In another specific embodiment, the third largest model can not only perform the image-text analysis task to generate the second matching degree / third matching degree, but also output whether the first candidate image / second candidate image matches the input text information and give the reason for the match or not.

[0146] According to an embodiment of the present disclosure, the second prompt information is also used to prompt the third largest model to perform an interpretation task; the method also includes: using the third largest model to perform an interpretation task on the input text information and at least one first candidate image based on the second prompt information, and generating at least one first interpretation information, the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, the first sub-interpretation information is used to interpret objects with the same semantics as the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics from the input text information and the first candidate image; or, using the third largest model to perform an interpretation task on the input text information and at least one second candidate image based on the second prompt information, and generating at least one second interpretation information, the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, the third sub-interpretation information is used to interpret objects with the same semantics as the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics from the input text information and the second candidate image.

[0147] The second prompt information prompts the third model to perform the interpretation task through the newly added interpretation prompt information. At this time, the information in the second prompt information used to prompt the third model to perform the graphic analysis task can be called graphic analysis prompt information.

[0148] For example, the image and text analysis prompt is the same as the second prompt above. The newly added explanation prompt may be: "After the ANSWER line, briefly explain how the candidate image meets or does not meet the text instructions. Important note: Only analyze based on text instructions and related images. Ignore elements unrelated to the text instructions. Do not introduce content beyond the text instructions." Among them, "Only analyze based on text instructions and related images. Ignore elements unrelated to the text instructions. Do not introduce content beyond the text instructions. Always start with the ANSWER line, and then provide explanations on new lines." is used to avoid hallucination output.

[0149] For example, the input text information is as follows: A woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her. The first candidate image / second candidate image is: A woman in a black shirt is cooking on the stove, and there is a black dog and a brown dog sitting next to her. At this time, the third largest model outputs ANSWER: [Yes], the second matching degree / third matching degree is 1, and the image includes a brown dog that is not in the text. Among them, "the image includes a brown dog that is not in the text" is the second sub-explanatory information in the first explanation information, or the fourth sub-explanatory information in the second explanation information. In another example, the output of the third largest model can be: [Yes], the second matching degree / third matching degree is 1, the image includes a brown dog that is not in the text, the image includes a black dog that exists in the text, and "the image includes a black dog that exists in the text" is the first sub-explanatory information in the first explanation information, or the third sub-explanatory information in the second explanation information.

[0150] At this point, it should be noted that for the prompt information "Analyze only based on text instructions and related images. Ignore elements unrelated to text instructions. Do not introduce content beyond text instructions", the "brown dog" not included in the input text information was ignored during the image and text analysis, and thus the answer of YES was obtained, but it was explained when explaining the task to facilitate user understanding.

[0151] In the embodiments of the present disclosure, in the process of verifying local semantics and global semantics, not only matching candidate images are returned, but also first interpretation information / second interpretation information can be generated to analyze inconsistent objects and details between images and texts, thereby improving the interpretability and transparency of the image search process and avoiding poor user experience caused by difficulty in understanding the success / failure of the image search task due to obtaining a simple similarity value.

[0152] Figure 8 A schematic diagram of a scenario in which the third large model according to an embodiment of the present disclosure performs graphic and text analysis tasks and interpretation tasks is schematically shown.

[0153] like Figure 8 As shown, the scene 800 includes two embodiments. In one embodiment, the third largest model M3 is used to perform the graphic analysis task T3 and the interpretation task T4 on the input text information 801 and the first candidate image 802 based on the second prompt information, and the second matching degree 804 and the first interpretation information 805 are obtained respectively. Since the first interpretation information 805 is an explanation of the difference between the first candidate image 802 and the input text information 801, each first candidate image corresponds to a first interpretation information. Therefore, after at least one target image is obtained by screening according to the first matching degree and the second matching degree 804, the first interpretation information of the target image can be obtained according to the correspondence between the first candidate image and the target image, thereby generating a search image sequence.

[0154] In another embodiment, the third large model M3 is used to perform the graphic analysis task T3 and the interpretation task T4 on the input text information 801 and the second candidate image 803 based on the second prompt information, and respectively obtain the third matching degree 807 and the second interpretation information 806. Similar to the previous embodiment, after at least one target image is screened using the third matching degree 807, the second interpretation information of the target image can be obtained according to the correspondence between the second candidate image and the target image, thereby generating a search image sequence.

[0155] According to an embodiment of the present disclosure, the method further includes: sequentially arranging at least one target image and first interpretation information into an image search sequence according to a degree of matching, and outputting the image search sequence; or, sequentially arranging at least one target image and second interpretation information into an image search sequence according to a degree of matching, and outputting the image search sequence.

[0156] The matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree. The sorting method according to the matching degree is referred to above and will not be repeated here.

[0157] The image search sequence can be arranged in order in the form of information pairs, such as a target image and the corresponding first interpretation information / second interpretation information form an information pair. When outputting the image search sequence, the first interpretation information / second interpretation information can be displayed near the target image.

[0158] In an embodiment of the present disclosure, the output image search sequence not only includes at least one target image searched, but also includes first explanation information / second explanation information of the searched target image, which makes it convenient for users to easily understand the success / failure of the image search task and its reasons based on the output image search sequence, and provides a better user experience.

[0159] According to an embodiment of the present disclosure, for operation S210, obtaining at least one first candidate image matching the input text information includes: determining text encoding features of the input text information; and determining at least one first candidate image based on the similarity between the text encoding features and image encoding features of each image in the image library.

[0160] For example, a text model can be used to encode input text information of a language modality to obtain text encoding features, such as a text vector; an image model can be used to encode an image of a visual modality to obtain image encoding features, such as an image vector.

[0161] Alternatively, a multimodal large model can be used to encode the input text information and images respectively to obtain text encoding features and image encoding features of each image in the image library. For example, a vision-language model (VLM) consisting of a visual encoder and a language encoder can be used to encode the input text information using the internal language encoder to obtain text encoding features; and to encode the image using the internal visual encoder to obtain image encoding features.

[0162] The similarity between the text encoding feature and the image encoding feature can be represented by the distance in the vector space, such as calculating the similarity by distance measurement methods such as cosine similarity and Euclidean distance. Then, at least one first candidate image matching the input text information can be preliminarily screened in the image library through a predetermined preliminary screening threshold.

[0163] In one embodiment, the image coding feature of each image in the image library may be encoded and stored in advance. When performing image search, the image coding feature of each image may be directly obtained and the similarity between the image coding feature and the text coding feature of the current input text information may be calculated.

[0164] For example, for an input text information T, the visual language model VLM is used to obtain the text encoding features of the input text information T, such as using the normalized text vector express, , d represents the dimension of the text vector, which is determined by the pre-trained visual language model VLM. The image encoding features of P images in the image library can be composed of the image matrix composed of normalized image vectors express, . Then, the text vector With the image matrix Perform a preliminary similarity calculation to obtain the similarity of each image to the input text information. The similarity is calculated using the following formula (1):

[0165] (1)

[0166] in, represents the inner product, Represents the similarity function, and the output result s is a vector of length P. , each element in the vector s represents the similarity between the corresponding image and the input text information. Sort the similarities from high to low to get a list of images that match the input text information. Then, N images can be selected as the first candidate images, denoted as .

[0167] In an embodiment of the present disclosure, at least one first candidate image is pre-screened based on the similarity between the text encoding features of the input text image and the image encoding features of each image in the image library. Thus, while ensuring the accuracy of image search, the amount of image processing of subsequent large models is reduced, resource usage is reduced, and the image search speed is improved.

[0168] To facilitate understanding of the present disclosure, a specific embodiment will be taken as an example to illustrate the image search process. Fig. 9 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown.

[0169] In the embodiment 900, the input text information T may be "a woman in a black shirt is cooking on a stove, with a black dog sitting next to her", and at least one first candidate image matching the input text information T obtained through the predetermined screening is .

[0170] The first model Reasoner is used to perform semantic analysis tasks and form conversion tasks based on Prompt1 (the first prompt information) and input text information T. After performing the first step of semantic analysis, M sub-input text information p is obtained. i , using a collection Representation. For example, the sub-input text information in the form of 4 declarative sentences is as follows: "1. There is a lady wearing a black shirt in the picture. 2. The lady in the picture is cooking. 3. There is a black dog in the picture. 4. The black dog is next to the lady in the picture." After performing the second step of the form conversion task, at least one question-answer pair is obtained as "1. Question: Is there a lady wearing a black shirt? | Yes; 2. Question: Is there a lady cooking? | Yes; 3. Question: Is there a black dog? | Yes; 4. Is there a black dog next to the lady? | Yes;". The generated M question-answer pairs can be a collection of question information and the collection of correct first answer information .

[0171] The execution process of the first model can be expressed by the following formula (2):

[0172] (2)

[0173] Using the second largest model to train at least one first candidate image Perform graphic and text analysis on at least one question information generated by the first large model, and generate second answer information for each question information. Afterwards, compare the first answer information with the second answer information for each question information. For example, for image 1 in the first candidate image, the comparison results of questions 1 to 4 are: different, same, same, different, and the number of question information with the same comparison results is 2; for image 2 in the first candidate image, the comparison results of questions 1 to 4 are: different, different, same, same, and the number of question information with the same comparison results is 2...For image N, the comparison results of questions 1 to 4 are: different, different, same, different, and the number of question information with the same comparison results is 1. In this embodiment, the comparison results are different and the same are respectively expressed as “ "and" "express.

[0174] The process of determining the second candidate image from the first candidate image can be expressed by the following formulas (3) to (5):

[0175] (3)

[0176] (4)

[0177] (5)

[0178] Among them, the second largest model Verifier is for the first candidate image and problem information Perform graphic analysis to obtain the second answer information , through the indicator function Determine the second answer information for the same question information j And the first answer information If they are the same, it is 1, otherwise it is 0. For each candidate image I, the number of the same first answer information and the same second answer information under M question information is combined to obtain the first matching degree . The vector representing the first matching degree, . According to Sort the values ​​in descending order, It means that after local semantic verification, at least one second candidate image is obtained. .

[0179] Use the third model Evaluator to evaluate at least one second candidate image based on Prompt2 (second prompt information) The image and text analysis is performed with the input text information T to generate a third matching degree for each second candidate image to evaluate the matching degree of each second candidate image with the input text information in terms of global semantics. Fig. 9 As shown, the second candidate image The third matching degree is 0, 1...1 respectively. Next to the " "and" "Sequence representation.

[0180] like Fig. 9 As shown, by using the third matching degree, the order of image 2 can be adjusted forward to obtain at least one target image , and feeds back the image search sequence as output to the user.

[0181] The process of determining the target image from the second candidate image can be expressed by the following formulas (6) to (8):

[0182] (6)

[0183] (7)

[0184] (8)

[0185] in, Indicates the third largest model Evaluator for the second candidate image The third matching degree obtained after global semantic verification with the input text information T is 1 (indicating compliance) or 0 (indicating non-compliance). A vector representing the third matching degree of all second candidate images, consisting of 0 or 1. represents the result of re-ranking the second candidate image after obtaining the third matching result through global semantic verification, The meaning is the same as the explanation part of formulas (3) to (5) above.

[0186] In addition, the third model Evaluator can also generate explanation information based on Prompt2 and output the explanation information.

[0187] The embodiments of the present disclosure perform multi-level semantic verification on images at the local semantic and global semantic levels. Local semantic verification ensures that the searched images meet the image search requirements in specific details, such as object color, shape and background; global semantic verification confirms the overall semantic relationship to ensure that the searched target image is not only similar to the input text information in terms of features, but also highly consistent with complex semantic relationships (such as spatial relationships between objects), thereby effectively improving the matching accuracy of complex semantic relationships and details, making the returned target image more compatible with the image search requirements, and the image search accuracy is higher. In addition, the output image search sequence includes the target image and explanation information, which makes it easier for users to understand the success / failure of the image search task and its reasons, and provides a better user experience. .

[0188] Fig.10 A block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown.

[0189] like Fig.10 As shown, the image search device 1000 includes: an acquisition module 1010 , a semantic analysis module 1020 , a graphic and text analysis module 1030 , and a determination module 1040 .

[0190] The acquisition module 1010 is used to acquire at least one first candidate image that is similar to input text information input by a user, wherein the input text information is used to represent the image search requirement of the user.

[0191] The semantic analysis module 1020 is used to perform semantic analysis on the input text information using the first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information.

[0192] The image and text analysis module 1030 is used to perform image and text analysis on at least one question information and at least one first candidate image using the second large model to generate second answer information for answering each question information.

[0193] The determination module 1040 is used to determine at least one target image matching the image search requirement from at least one first candidate image according to the comparison result between at least one first answer information and at least one second answer information.

[0194] According to an embodiment of the present disclosure, the semantic analysis module 1020 includes: a first acquisition submodule, a semantic analysis submodule, and a form conversion submodule.

[0195] The first acquisition submodule is used to acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task.

[0196] The semantic analysis submodule is used to perform a semantic analysis task based on the first prompt information and the input text information using the first large model to extract at least one sub-input text information from the input text information.

[0197] The form conversion submodule is used to use the first large model to perform a form conversion task on each sub-input text information based on the first prompt information to convert the sub-input text information into a question-answer pair.

[0198] According to an embodiment of the present disclosure, the semantic analysis submodule includes:

[0199] The first semantic analysis unit is used to perform a semantic analysis task based on the first prompt information and the input text information using the first large model, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; wherein the semantic dimensions include at least one of the following: color, shape, and background.

[0200] According to an embodiment of the present disclosure, the acquisition submodule includes: a first combination unit, which is used to combine the semantic splitting example and the form conversion example to obtain the first prompt information.

[0201] According to an embodiment of the present disclosure, the acquisition submodule includes: a first example determination unit and a second combination unit.

[0202] The first example determination unit is used to determine, according to the scene type of the input text information, a semantic split example and a form conversion example that match the scene type.

[0203] The second combining unit is used to combine the semantic splitting examples and the form conversion examples that match the scene type to obtain the first prompt information.

[0204] According to an embodiment of the present disclosure, the acquisition submodule includes: a second example determination unit and a third combination unit.

[0205] A second example determination unit is used to determine at least one semantic split example and at least one form conversion example according to the text complexity of the input text information; and

[0206] The third combining unit is used to combine at least one semantic splitting example and at least one form conversion example to obtain first prompt information.

[0207] According to an embodiment of the present disclosure, the determination module 1040 includes a first determination submodule, a second determination submodule and a first screening submodule.

[0208] The first determination submodule is used to determine, for each first candidate image, a comparison result of first answer information and second answer information for answering the same question information, wherein the comparison result includes the same or different.

[0209] The second determination submodule is used to determine the first matching degree of each first candidate image according to the number of question information with the same comparison result.

[0210] The first screening submodule is used to screen out at least one target image matching the image search requirement from at least one first candidate image according to a first matching degree.

[0211] According to an embodiment of the present disclosure, the determination module 1040 further includes: a first image and text analysis submodule and a second screening submodule.

[0212] The first image-text analysis submodule is used to perform image-text analysis on the input text information and at least one first candidate image using the third large model to generate at least one second matching degree, wherein the second matching degree is used to characterize whether the semantics of the first candidate image and the input text information match.

[0213] The second screening submodule is used to screen out at least one target image matching the image search requirement from at least one first candidate image according to the first matching degree and the second matching degree.

[0214] According to an embodiment of the present disclosure, the determination module 1040 further includes: a third screening submodule, a second graphic analysis submodule and a fourth screening submodule.

[0215] The third screening submodule is used to screen out at least one second candidate image from at least one first candidate image according to the first matching degree.

[0216] The second image-text analysis submodule is used to perform image-text analysis on the input text information and at least one second candidate image using the third large model to generate at least one third matching degree, wherein the third matching degree is used to characterize whether the semantics of the second candidate image and the input text information match.

[0217] The fourth screening submodule is used to screen out at least one target image matching the image search requirement from at least one second candidate image according to the third matching degree.

[0218] According to an embodiment of the present disclosure, the determination module 1040 further includes: a second acquisition submodule and a third graphic and text analysis submodule.

[0219] The second acquisition submodule is used to obtain second prompt information, wherein the second prompt information is used to prompt the third model to perform a graphic analysis task.

[0220] The third image-text analysis submodule is used to perform image-text analysis tasks on the input text information and at least one first candidate image based on the second prompt information using the third largest model to generate at least one second matching degree; or to perform image-text analysis tasks on the input text information and at least one second candidate image based on the second prompt information using the third largest model to generate at least one third matching degree.

[0221] According to an embodiment of the present disclosure, the image search device 1000 also includes: a first output module, used to sequentially organize at least one target image into an image search sequence according to the matching degree, and return the image search sequence to the user, wherein the matching degree includes at least one of the following: a first matching degree, a second matching degree, and a third matching degree.

[0222] According to an embodiment of the present disclosure, the second prompt information is also used to prompt the third largest model to perform the interpretation task; the determination module 1040 also includes: a first interpretation submodule or a second interpretation submodule.

[0223] The first interpretation submodule is used to perform an interpretation task on the input text information and at least one first candidate image based on the second prompt information using the third large model, and generate at least one first interpretation information, wherein the first interpretation information is used to interpret objects with the same and / or different semantics in the input text information and the first candidate image.

[0224] The second interpretation submodule is used to perform an interpretation task on the input text information and at least one second candidate image based on the second prompt information using the third large model, and generate at least one second interpretation information, wherein the second interpretation information is used to interpret objects with the same and / or different semantics in the input text information and the second candidate image.

[0225] According to an embodiment of the present disclosure, the image search apparatus 1000 further includes: a second output module or a third output module.

[0226] The second output module is used to sequentially combine at least one target image and the first interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

[0227] The third output module is used to sequentially combine the at least one target image and the second interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

[0228] According to an embodiment of the present disclosure, the acquisition module 1010 includes: a feature determination submodule and an image determination submodule.

[0229] The feature determination submodule is used to determine the text coding feature of the input text information. The image determination submodule is used to determine at least one first candidate image according to the similarity between the text coding feature and the image coding feature of each image in the image library.

[0230] Fig.11 The structure block diagram of the intelligent agent of artificial intelligence according to the embodiment of the present disclosure is schematically shown. In the embodiment of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig.11 As shown, the AI ​​agent 1100 may include five core modules: an input module 1110 , a control module 1120 , a storage module 1130 , a calculation module 1140 and an output module 1150 .

[0231] The input module 1110 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 1100. The input module 1110 is the primary link for the AI ​​agent 1100 to interact with the outside world. It enables the AI ​​agent 1100 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0232] In an example, the input module 1110 may input the input text information described above.

[0233] In the example, the control module 1120 is the core support for the AI ​​agent 1100 to handle complex tasks. The control module 1120 can execute the image search method described above.

[0234] In the example, the control module 1120 will continuously interact with the storage module 1130, the computing module 1140 and / or the output module 1150 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 1120 acts as a single initiator to initiate communication with the storage module 1130, the computing module 1140 and / or the output module 1150, and there is no communication coupling between the storage module 1130, the computing module 1140 and the output module 1150.

[0235] In the example, the performance of the control module 1120 may be closely related to the large model on which the AI ​​agent 1100 is based. In order to give full play to the capabilities of the large model, the internal structure of the control module 1120 may be designed to be highly configurable and extensible, so as to cope with various types of tasks and requirements in real scenarios, such as the semantic analysis tasks, form conversion tasks, graphic analysis tasks, interpretation tasks, etc. mentioned above.

[0236] The storage module 1130 may be responsible for memorizing information such as historical conversations, event flows, etc. The prompt information and multiple seed input text information, question-answer pairs, etc. as mentioned above may be included in the storage module 1130 .

[0237] The operation module 1140 can be regarded as a predefined tool library. As described above, the controls for text encoding and image encoding can be included in the operation module 1140.

[0238] In an example, the output module 1150 may output at least one target image described above.

[0239] The AI ​​agent 1100 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0240] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0241] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0242] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0243] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0244] Fig.12 A block diagram of an electronic device 1200 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0245] like Fig.12As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage unit 1208 into a random access memory (RAM) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0246] A number of components in the device 1200 are connected to an input / output (I / O) interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0247] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1201 performs the various methods and processes described above, such as the image search method. For example, in some embodiments, the image search method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the image search method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform the image search method in any other appropriate manner (e.g., by means of firmware).

[0248] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0249] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0250] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0251] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0252] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0253] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0254] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0255] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image search method, comprising: Acquire at least one first candidate image matching the input text information, wherein the input text information is used to represent the image search requirement; Performing semantic analysis on the input text information using the first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; Using a second large model to perform graphic and text analysis on at least one of the question information and at least one of the first candidate images, to generate second answer information for answering each of the question information; and At least one target image matching the image search requirement is determined from at least one first candidate image according to a comparison result between at least one of the first answer information and at least one of the second answer information.

2. The method according to claim 1, wherein: The using the first large model to perform semantic analysis on the input text information to generate at least one question-answer pair includes: Acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; Using the first large model to perform the semantic analysis task based on the first prompt information and the input text information, so as to extract at least one sub-input text information from the input text information; and The first large model is used to perform the form conversion task on each of the sub-input text information based on the first prompt information to convert the sub-input text information into the question-answer pair.

3. The method according to claim 2, wherein: The using the first large model to perform the semantic analysis task based on the first prompt information and the input text information to extract at least one sub-input text information from the input text information includes: Using the first large model to perform the semantic analysis task based on the first prompt information and the input text information, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; The semantic dimension includes at least one of the following: color, shape, and background.

4. The method according to claim 2 or 3, wherein: The obtaining of the first prompt information comprises: The first prompt information is obtained by combining the semantic splitting example and the form conversion example.

5. The method according to any one of claims 2 to 4, wherein: The obtaining of the first prompt information includes: According to the scene type of the input text information, determining a semantic splitting example and a form conversion example matching the scene type; and The first prompt information is obtained by combining the semantic splitting examples and the form conversion examples that match the scene type.

6. The method according to any one of claims 2 to 5, wherein: The obtaining of the first prompt information includes: Determining at least one of the semantic splitting examples and at least one of the form conversion examples according to the text complexity of the input text information; and The first prompt information is obtained by combining at least one of the semantic splitting examples and at least one of the form conversion examples.

7. The method according to claim 1, wherein: The determining, based on the comparison result between at least one of the first answer information and at least one of the second answer information, at least one target image matching the image search requirement from at least one of the first candidate images comprises: For each of the first candidate images, determining a comparison result of first answer information and second answer information for answering the same question information; Determining a first matching degree between each of the first candidate images and the input text information according to the comparison result; and At least one of the target images matching the image search requirement is screened out from at least one of the first candidate images according to the first matching degree.

8. The method according to claim 7, further comprising: Performing a graphic analysis on the input text information and at least one of the first candidate images using a third model to generate at least one second matching degree, wherein the second matching degree is used to represent a matching degree between the semantics of the first candidate image and the input text information; and At least one of the target images matching the image search requirement is screened out from at least one of the first candidate images according to the first matching degree and the second matching degree.

9. The method according to claim 7, further comprising: Screening out at least one second candidate image from at least one of the first candidate images according to the first matching degree; Performing a graphic analysis on the input text information and at least one of the second candidate images using a third large model to generate at least one third matching degree, wherein the third matching degree is used to represent a matching degree between the semantics of the second candidate image and the input text information; as well as At least one of the target images matching the image search requirement is screened out from at least one of the second candidate images according to the third matching degree.

10. The method according to claim 8 or 9, further comprising: Acquire second prompt information, wherein the second prompt information is used to prompt the third model to perform a graphic and text analysis task; Using a third large model to perform the image-text analysis task on the input text information and at least one of the first candidate images based on the second prompt information to generate at least one second matching degree; or using a third large model to perform the image-text analysis task on the input text information and at least one of the second candidate images based on the second prompt information to generate at least one third matching degree.

11. The method according to any one of claims 7 to 10, further comprising: At least one of the target images is sequentially organized into an image search sequence according to a matching degree, and the image search sequence is output, wherein the matching degree includes at least one of the following: the first matching degree, the second matching degree, and the third matching degree.

12. The method according to claim 10, wherein: The second prompt information is also used to prompt the third largest model to perform an interpretation task; the method further includes: Using a third large model to perform the interpretation task on the input text information and at least one of the first candidate images based on the second prompt information, generating at least one first interpretation information, wherein the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, the first sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics from those in the input text information and the first candidate image; or, The interpretation task is performed on the input text information and at least one of the second candidate images based on the second prompt information using a third large model to generate at least one second interpretation information, wherein the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, wherein the third sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics as those in the input text information and the second candidate image.

13. The method according to claim 12, further comprising: Sequentially composing at least one of the target images and the first interpretation information into an image search sequence according to the matching degree, and outputting the image search sequence; or, At least one of the target images and the second interpretation information are sequentially combined into an image search sequence according to the matching degree, and the image search sequence is output.

14. The method according to any one of claims 1 to 13, wherein: The step of acquiring at least one first candidate image matching the input text information comprises: Determining text encoding features of the input text information; and At least one of the first candidate images is determined according to the similarity between the text encoding feature and the image encoding feature of each image in the image library.

15. An image search device, comprising: An acquisition module, used to acquire at least one first candidate image similar to input text information input by a user, wherein the input text information is used to represent the image search requirement of the user; A semantic analysis module, configured to perform semantic analysis on the input text information using a first large model to generate at least one question-answer pair, wherein the question-answer pair includes question information and first answer information extracted from the input text information; A graphic and text analysis module, configured to use a second large model to perform graphic and text analysis on at least one of the question information and at least one of the first candidate images, and generate second answer information for answering each of the question information; and A determination module is used to determine at least one target image matching the image search requirement from at least one of the first candidate images based on a comparison result between at least one of the first answer information and at least one of the second answer information.

16. The device according to claim 15, wherein: The semantic analysis module comprises: A first acquisition submodule is used to acquire first prompt information, wherein the first prompt information is used to prompt the first large model to perform a semantic analysis task and a form conversion task; a semantic analysis submodule, configured to use the first large model to perform the semantic analysis task based on the first prompt information and the input text information, so as to extract at least one sub-input text information from the input text information; and A form conversion submodule is used to use the first large model to perform the form conversion task on each of the sub-input text information based on the first prompt information, so as to convert the sub-input text information into the question-answer pair.

17. The device according to claim 16, wherein: The semantic analysis submodule includes: a first semantic analysis unit, configured to perform the semantic analysis task based on the first prompt information and the input text information by using the first large model, so as to extract at least one sub-input text information from the input text information based on multiple semantic dimensions; The semantic dimension includes at least one of the following: color, shape, and background.

18. The device according to claim 16 or 17, wherein: The acquisition submodule includes: The first combining unit is used to combine the semantic splitting example and the form conversion example to obtain the first prompt information.

19. The device according to any one of claims 16 to 18, wherein: The acquisition submodule includes: A first example determination unit is configured to determine, according to the scene type of the input text information, a semantic split example and a form conversion example matching the scene type; and The second combining unit is used to combine the first prompt information according to the semantic splitting examples and the form conversion examples matching the scene type.

20. The device according to any one of claims 16 to 19, wherein: The acquisition submodule includes: A second example determination unit is configured to determine at least one of the semantic splitting examples and at least one of the form conversion examples according to the text complexity of the input text information; and The third combining unit is used to combine at least one of the semantic splitting examples and at least one of the form conversion examples to obtain the first prompt information.

21. The device according to claim 15, wherein: The determination module comprises: A first determination submodule, used for determining, for each of the first candidate images, a comparison result between first answer information and second answer information for answering the same question information; A second determination submodule, configured to determine a first matching degree of each of the first candidate images according to the comparison result; and The first screening submodule is used to screen out at least one target image matching the image search requirement from at least one of the first candidate images according to the first matching degree.

22. The apparatus according to claim 21, wherein the determining module further comprises: a first image-text analysis submodule, configured to use a third large model to perform image-text analysis on the input text information and at least one of the first candidate images to generate at least one second matching degree, wherein the second matching degree is used to indicate whether the semantics of the first candidate image and the input text information match; and The second screening submodule is used to screen out at least one target image matching the image search requirement from at least one of the first candidate images according to the first matching degree and the second matching degree.

23. The apparatus according to claim 21, wherein the determining module further comprises: A third screening submodule, configured to screen out at least one second candidate image from at least one of the first candidate images according to the first matching degree; a second image-text analysis submodule, configured to use a third large model to perform image-text analysis on the input text information and at least one of the second candidate images, and generate at least one third matching degree, wherein the third matching degree is used to indicate whether the semantics of the second candidate image and the input text information match; as well as The fourth screening submodule is used to screen out at least one target image matching the image search requirement from at least one of the second candidate images according to the third matching degree.

24. The apparatus according to claim 22 or 23, wherein the determining module further comprises: A second acquisition submodule is used to acquire second prompt information, wherein the second prompt information is used to prompt the third model to perform a graphic analysis task; The third image-text analysis submodule is used to use the third largest model to perform the image-text analysis task on the input text information and at least one of the first candidate images based on the second prompt information to generate at least one second matching degree; or to use the third largest model to perform the image-text analysis task on the input text information and at least one of the second candidate images based on the second prompt information to generate at least one third matching degree.

25. The device according to any one of claims 21 to 24, wherein the image search device further comprises: The first output module is used to sequentially organize at least one of the target images into an image search sequence according to a matching degree, and return the image search sequence to the user, wherein the matching degree includes at least one of the following: the first matching degree, the second matching degree, and the third matching degree.

26. The device according to claim 24, wherein: The second prompt information is also used to prompt the third model to perform an interpretation task; The determination module also includes: a first interpretation submodule, configured to perform the interpretation task on the input text information and at least one of the first candidate images based on the second prompt information using a third large model, and generate at least one first interpretation information, wherein the first interpretation information includes first sub-interpretation information and / or second sub-interpretation information, wherein the first sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the first candidate image, and the second sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the first candidate image; or, The second interpretation submodule is used to perform the interpretation task on the input text information and at least one of the second candidate images based on the second prompt information using a third large model to generate at least one second interpretation information, wherein the second interpretation information includes third sub-interpretation information and / or fourth sub-interpretation information, wherein the third sub-interpretation information is used to interpret objects with the same semantics as those in the input text information and the second candidate image, and the fourth sub-interpretation information is used to interpret objects with different semantics than those in the input text information and the second candidate image.

27. The apparatus according to claim 26, wherein the image search apparatus further comprises: a second output module, configured to sequentially combine at least one of the target images and the first interpretation information into an image search sequence according to a matching degree, and return the image search sequence to the user; or, A third output module is used to sequentially combine at least one of the target images and the second interpretation information into an image search sequence according to the matching degree, and return the image search sequence to the user.

28. The device according to any one of claims 15 to 27, wherein: The acquisition module comprises: a feature determination submodule, used to determine text encoding features of the input text information; and The image determination submodule is used to determine at least one of the first candidate images according to the similarity between the text encoding feature and the image encoding feature of each image in the image library.

29. An agent configured to execute the method according to any one of claims 1 to 14.

30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 14.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

32. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images