Image search method and device

Abstract

The present invention discloses an image search method and device, which relate to the field of artificial intelligence technology, and aim to solve the problem of single modal input limitation and insufficient understanding of search intent of traditional image search methods, including: first, obtaining a multimodal search demand instruction input by a user, which may include text requirements and image requirements. When the user's intention is fully expressed, these instructions are input into a pre-trained image search agent for task splitting to obtain at least one task to be executed. Subsequently, the image search agent calls the target model corresponding to each task to execute these tasks and obtain the execution results. Finally, based on these results, the target image is extracted from a preset offline image library as the image search result corresponding to the multimodal search demand instruction. Such a design can improve the accuracy of image search and user satisfaction, and is suitable for the fields of multimodal information retrieval and intelligent recommendation systems.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image search method and device. Background Art

[0002] With the rapid development of information technology, image search has become an important way for people to obtain information. However, traditional image search methods are mostly based on single text or image input, which is difficult to meet users' diverse search needs. Summary of the invention

[0003] The object of the present invention is to provide an image search method and device.

[0004] In a first aspect, an embodiment of the present invention provides an image search method, comprising:

[0005] Acquire at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement;

[0006] In a case where the at least one multimodal search requirement instruction represents a complete user intention, inputting the multimodal search requirement instruction into a pre-trained image search agent;

[0007] Using the image search agent to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction;

[0008] Based on the image search agent, the target model corresponding to the at least one task to be performed is called to perform the at least one task to be performed, and a corresponding task execution result is obtained;

[0009] Based on the task execution result, a target image is extracted from a preset offline image library as an image search result corresponding to the multimodal search requirement instruction.

[0010] In a second aspect, an embodiment of the present invention provides an image search device, including:

[0011] An acquisition module, used to acquire at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement;

[0012] An execution module is used to input the multimodal search requirement instruction into a pre-trained image search agent when the at least one multimodal search requirement instruction represents a complete user intention; use the image search agent to split the multimodal search requirement instruction into tasks to obtain at least one task to be executed corresponding to the multimodal search requirement instruction; based on the image search agent, call the target model corresponding to the at least one task to be executed to execute the at least one task to be executed, and obtain the corresponding task execution result;

[0013] The output module is used to extract the target image from a preset offline image library as the image search result corresponding to the multimodal search requirement instruction based on the task execution result.

[0014] Compared with the prior art, the beneficial effects provided by the present invention include: adopting an image search method and device disclosed in the present invention, by obtaining multimodal search demand instructions input by the user, using an image search agent to perform task splitting and model calling, and extracting target images from a preset offline image library as search results. Such a design effectively supports multimodal search needs and improves search efficiency and user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.

[0016] Figure 1 A schematic diagram of the steps of the image search method provided by an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the overall framework of an image search system provided by an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of an image to be searched provided by an embodiment of the present invention;

[0019] Figure 4 A schematic diagram of the workflow of an image search agent provided by an embodiment of the present invention;

[0020] Figure 5 A schematic diagram of the steps of task splitting provided in an embodiment of the present invention;

[0021] Figure 6 A schematic diagram of the overall framework of an offline warehousing module provided in an embodiment of the present invention;

[0022] Figure 7 A schematic block diagram of the structure of an image search device provided by an embodiment of the present invention;

[0023] Figure 8 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0025] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.

[0026] In order to solve the technical problems in the aforementioned background technology, Figure 1 This is a flow chart of an image search method provided in an embodiment of the present disclosure. The image search method is introduced in detail below.

[0027] Step S201, obtaining at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement;

[0028] Step S202, when the at least one multimodal search requirement instruction represents a complete user intention, inputting the multimodal search requirement instruction into a pre-trained image search agent;

[0029] Step S203, using the image search agent to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction;

[0030] Step S204, based on the image search agent, calling the target model corresponding to the at least one task to be executed to execute the at least one task to be executed, and obtaining a corresponding task execution result;

[0031] Step S205 , based on the task execution result, extracting a target image from a preset offline image library as an image search result corresponding to the multimodal search requirement instruction.

[0032] In an embodiment of the present invention, illustratively, a user sends a search request to a server through a web interface or a mobile application. The request includes a multimodal search requirement instruction, that is, the user provides both a text description and a picture as search conditions. For example, the user uploads a picture containing a beach and a surfboard, and enters the text "Find similar surfing scenes" in the search box.

[0033] After receiving the user input, the server first determines whether the user's intention is clear and complete. In this example, the user's intention is clear, that is, to search for surfing scenes similar to the uploaded image. The server takes the text and image provided by the user as input and passes it to the pre-trained image search agent.

[0034] After receiving the input, the image search agent begins to analyze and split the task. In this example, the agent needs to identify the objects and scenes in the image and prepare to use this information to search.

[0035] First call: The agent first calls the object detection model to identify objects (such as surfboards, waves, etc.) and scenes (such as beaches) in the image.

[0036] Non-first call: Depending on the results of target detection, the agent may also need to call other models to obtain more detailed information, such as using an OCR model to recognize text descriptions in the image (if any).

[0037] It should be understood that the agent in the embodiment of the present invention will autonomously decide whether to perform the task decomposition, execution, and observation steps again based on the results of the previous round. A task requires multiple cycles to get the result, and the number of rounds to execute and whether to end are all autonomous decisions made by the image search agent without human intervention. For problems that need to be solved through multiple rounds of cycles, one cycle here includes three steps: task decomposition-calling model-observing results, and multiple such cycles will be used throughout the life cycle.

[0038] The image search agent calls the corresponding target model in turn to perform tasks according to the split task order. The agent first calls the target detection model to return the object and scene information detected in the image.

[0039] Then, based on this information, the agent calls the vector retrieval model to prepare to search for similar images in the offline image library.

[0040] After obtaining the task execution results, the server uses these results to retrieve similar images from the offline image library. The server performs a search in the offline image library based on the object and scene information returned by the target detection model and the query vector generated by the vector retrieval model. The offline image library uses Elasticsearch and vector databases (such as Milvus) to store the text labels and vector information of the images. The server uses these indexes to quickly retrieve images similar to the target image. The server returns the retrieved similar images to the user as search results. The user receives the search results returned by the server, which include multiple surfing scene images similar to the uploaded image.

[0041] In the embodiments of the present invention, the following implementation modes are also provided.

[0042] Receiving a user's feedback multimodal search demand instruction for the image search results;

[0043] Based on the feedback multimodal search demand instruction, calling the image search agent to perform image retrieval in combination with the image search results to obtain candidate image search results;

[0044] When receiving the user's feedback multimodal search requirement instruction for the candidate image search results, the image search agent is called to perform image retrieval in combination with the candidate image search results.

[0045] In an embodiment of the present invention, illustratively, after the server returns the initial image search results to the user, the user browses the results but may not find a completely satisfactory picture. Therefore, the user provides further search instructions through the interface, including text description, picture adjustment, or selection of specific search results as reference. After viewing the surfing scene pictures returned by the server, the user finds that the background of most pictures is daytime, but the user prefers surfing scenes at night. Therefore, the user enters the text "surfing at night" on the interface and uploads an example picture of surfing at night. After receiving these feedback multimodal search demand instructions, the server prepares for the next step of the search.

[0046] After receiving the user's feedback instructions, the server calls the image search agent and combines the new instructions provided by the user with the previous search results to perform more detailed image retrieval. The image search agent first analyzes the text description of "night surfing" and the sample pictures provided by the user. The agent identifies the key elements in the sample pictures: night, waves, surfboards, and possible lighting effects. Then, the agent calls the corresponding model (such as the target detection model, image vectorization model) to process the new search instructions and sample pictures, and retrieves more relevant pictures in the offline image library based on this information. After the retrieval is completed, the server returns the new candidate image search results to the user.

[0047] After viewing the candidate image search results, the user may still be dissatisfied or want to further refine the search criteria. Therefore, the user provides feedback multimodal search demand instructions again. After browsing the candidate night surfing scene pictures, the user finds that many of the pictures show surfers wearing long-sleeved surfing suits, but the user prefers to find pictures wearing short-sleeved surfing suits. Therefore, the user selects several pictures of people wearing short-sleeved surfing suits on the interface as references and enters the text "short-sleeved surfing suit". After receiving these new feedback instructions, the server calls the image search agent again. The agent further adjusts the search strategy based on the reference pictures selected by the user and the new text description, and retrieves pictures that better meet the user's needs in the offline image library. After the retrieval is completed, the server returns the final image search results to the user. In this way, through multiple feedback and iterations, the server can gradually narrow the search scope and provide more accurate image search results.

[0048] In an embodiment of the present invention, the at least one multimodal search requirement instruction includes a first multimodal search requirement instruction and a second multimodal search requirement instruction, and the obtaining of at least one multimodal search requirement instruction input by a user may be implemented through the following example execution.

[0049] Obtaining the first multimodal search requirement instruction;

[0050] Parsing the first multimodal search requirement instruction to obtain a first parsing result;

[0051] Determining whether the first parsing result constitutes a complete intent requirement;

[0052] If so, determining that the first multimodal search requirement instruction represents a complete user intention;

[0053] If not, generating a feedback interaction instruction based on the first multimodal search requirement instruction;

[0054] Acquire the second multimodal search requirement instruction input by the user in response to the feedback interaction instruction;

[0055] Parsing the second multimodal search requirement instruction to obtain a second parsing result;

[0056] Determining whether the first analysis result combined with the second analysis result constitutes a complete intention requirement;

[0057] If so, determining that the first multimodal search requirement instruction combined with the second multimodal search requirement instruction represents a complete user intention;

[0058] If not, the step of generating feedback interaction instructions is repeated until a multimodal search demand instruction representing the complete user intention is obtained.

[0059] In an embodiment of the present invention, illustratively, a user sends an initial search request to a server through a search interface, and the request includes a first multimodal search requirement instruction, which may be a text description and a picture, or only one of them. The user uploads a picture containing a mountain view and enters the text "similar mountain pictures" in the search box. The server receives this request and uses the text and picture as the first multimodal search requirement instruction.

[0060] After receiving the first multimodal search demand instruction, the server begins to parse the instruction and understand the user's search intent. The server performs image analysis on the uploaded image to identify key elements in the image (such as mountains, clouds, etc.), and performs natural language processing on the text description to extract keywords (such as "similar" and "mountain").

[0061] The server determines whether the user's search intent is complete based on the first parsed result. If the text and image information is clear enough, the intent is complete; if the information is insufficient or ambiguous, the intent is incomplete. In this example, because the user provided a clear mountain image and text description, the server determines that the first parsed result constitutes a complete intent requirement.

[0062] If the first parsing result is incomplete, the server needs to generate a feedback interaction instruction to prompt the user to provide additional information. If the user only uploads a picture without entering a text description, the server may not accurately understand what type of mountain the user wants to search for (such as winter snow scene, mountain at sunrise, etc.). Therefore, the server generates a feedback interaction instruction to prompt the user to enter more text description.

[0063] After receiving the feedback interaction instruction from the server, the user supplements the search information according to the prompt, and the server obtains this supplementary information as the second multimodal search demand instruction. According to the prompt of the server, the user enters "mountains under winter snow scene" in the search box and confirms the previous picture. The server obtains this new information as the second multimodal search demand instruction.

[0064] The server parses the second multimodal search demand instruction to further understand the user's search intent. The server performs natural language processing on the new text description, extracts keywords (such as "winter" and "snow scene"), and combines the analysis results of the previous pictures to form a more complete search intent.

[0065] The server combines the first parsing result and the second parsing result to determine whether the user's search intention is complete. In this example, since the user adds a clear text description, the server determines that the first parsing result and the second parsing result together constitute a complete search intention.

[0066] If the parsing results are incomplete, the server will repeatedly generate feedback interaction instructions and interact with the user multiple times until a multimodal search requirement instruction that represents the complete user intent is obtained. If the search information provided by the user is incomplete or ambiguous multiple times in a row, the server will continue to give feedback interaction instructions to guide the user to gradually clarify the search intent.

[0067] In an embodiment of the present invention, the use of the image search agent to split the multimodal search demand instruction into tasks to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction can be implemented through the following examples.

[0068] A pre-trained general task classification large language model is called to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction.

[0069] In an embodiment of the present invention, exemplarily, after receiving the multimodal search demand instruction from the user, the server prepares to split the instruction into tasks. In order to accurately split the tasks, the server calls a pre-trained general task classification large language model (Large Language Model, LLM for short). This LLM has powerful language understanding and generation capabilities, as well as task classification functions. The server passes the multimodal search demand instruction (including text descriptions and pictures) as input to the LLM. The LLM begins to analyze the instruction, understand the user's search intent, and prepare to split the tasks.

[0070] After receiving the multimodal search demand instruction, LLM starts to split the task of this instruction. The purpose of splitting is to break down complex search requirements into a series of more specific and executable small tasks. Suppose the user's multimodal search demand instruction is: "Please help me find a mountain similar to the one in this picture, but with a view at sunrise." The user also uploaded a picture containing a mountain. LLM first analyzes this instruction and identifies the key information: the mountain in the picture and the view at sunrise. Then, LLM splits this complex search demand into the following tasks to be executed:

[0071] Object detection task: Identify the mountains in the uploaded image and determine their location and features.

[0072] Scene recognition task: Since the user requested the scenery at sunrise, LLM will call the scene recognition model to identify features related to sunrise, such as the position of the sun, the direction of light, etc.

[0073] Image retrieval task: Based on the results of object detection and scene recognition, retrieve pictures with similar features in the image library.

[0074] After task splitting, LLM generates one or more lists of tasks to be performed. These tasks will serve as guidance for the subsequent search process. In this example, LLM generates three tasks to be performed: object detection task, scene recognition task, and image retrieval task. After receiving these tasks, the server will execute them in sequence to complete the user's search needs. By calling the pre-trained general task classification large language model for task splitting, the server can accurately understand the user's search intent and decompose complex search needs into a series of executable small tasks. This provides clear guidance for the subsequent search process and improves the accuracy and efficiency of the search.

[0075] In an embodiment of the present invention, the image search agent calls the target models corresponding to the at least one task to be performed to execute the at least one task to be performed and obtains the corresponding task execution result, which can be implemented through the following examples.

[0076] Obtaining a task prompt word corresponding to the at least one task to be performed;

[0077] Determine, according to the task prompt word, a model list and a retrieval interface corresponding to the at least one task to be performed, and call a target model corresponding to the at least one task to be performed from the model list;

[0078] The at least one task to be executed is executed in combination with the retrieval interface and the target model to obtain a corresponding task execution result.

[0079] In an embodiment of the present invention, exemplarily, after the server splits the multimodal search demand instruction into tasks and obtains at least one task to be executed, it is necessary to obtain corresponding task prompt words for each task to be executed. These task prompt words will be used to guide the selection and calling of subsequent models. Assume that after the task splitting, the server obtains two tasks to be executed: one is a target detection task for identifying peaks in a picture; the other is an image retrieval task for retrieving pictures similar to the identified peaks in the image library. For the target detection task, the server obtains the corresponding task prompt words from a predefined task template or knowledge base, such as "identify the peaks in the picture and return their positions and features". For the image retrieval task, the server also obtains the corresponding task prompt words, such as "retrieve similar pictures in the image library based on the provided peak features and positions".

[0080] Based on the task prompts obtained, the server determines the model list that should be called for each task to be executed and the retrieval interface that may be used. Then call the appropriate target model from the model list. For the target detection task, the server determines that a target detection model needs to be called based on the task prompt "identify the peaks in the image and return their positions and features". Assuming that the server has a list of multiple target detection models, it will select the most appropriate model to call based on the task requirements. For the image retrieval task, the server determines that an image retrieval model needs to be called and the retrieval interface of the image library needs to be used based on the task prompt "retrieve similar images in the image library based on the provided peak features and positions". The server calls the image retrieval model from the model list and is ready to use the retrieval interface of the image library.

[0081] The server combines the previously determined retrieval interface and target model to start executing the tasks to be executed. After each task is completed, the server will collect the corresponding task execution results. For the target detection task, the server takes the picture uploaded by the user as input and calls the target detection model to identify and extract features of the peak. After the model is completed, the server obtains the location and feature information of the peak as the task execution result. For the image retrieval task, the server takes the peak features and location information extracted by the target detection model as input and calls the image retrieval model through the retrieval interface of the image library. The model searches for pictures similar to the input features in the image library and returns the retrieval results. The server uses these retrieval results as the task execution results.

[0082] Through the above steps, the server successfully determined the appropriate model and retrieval interface for each task to be executed, successfully executed the task, and obtained the corresponding task execution results. These results will provide important reference and basis for subsequent image search results.

[0083] In an embodiment of the present invention, the model list includes a picture-text multimodal model, a picture vectorization model, an open vocabulary target detection model, a target object recognition model, a text extraction model, and a key point recognition model;

[0084] The retrieval interface includes an external image search interface, a text retrieval interface, a vector retrieval interface, and a sorting interface.

[0085] In an embodiment of the present invention, exemplarily, when the server performs a multimodal image search task, it selects a suitable model from a predefined model list according to the task requirements, and uses the corresponding retrieval interface to perform the search. These model lists and retrieval interfaces are pre-configured to support different types of search tasks. The server maintains a model list, including a multimodal model of images and texts, a picture vectorization model, an open vocabulary target detection model, a target object recognition model, a text extraction model (such as an OCR model), a key point recognition model, etc. Each model has its specific functions and applicable scenarios. At the same time, the server also maintains a set of retrieval interfaces, including an external image search interface (such as an API that calls external search engines such as Baidu and Google), a text retrieval interface (used to retrieve information in a text database or index), a vector retrieval interface (used to retrieve images corresponding to similar vectors in a vector database) and a sorting interface (used to sort search results).

[0086] Suppose a user uploads a picture of a mountain and sunrise, and enters a text description "Please find pictures of sunrise mountains similar to this one". The server needs to analyze this multimodal search requirement, select a suitable model from the model list, and use the corresponding retrieval interface to perform the search. Task analysis: The server first analyzes the user's multimodal search requirement, identifies the object detection task (identifying the mountain in the picture), the scene recognition task (confirming the scenery at sunrise), and the image retrieval task (retrieve similar pictures in the image library) to be performed. Model selection: For the object detection task, the server selects the open vocabulary object detection model from the model list. This model can recognize a variety of objects in the picture, including mountains. For the scene recognition task, since scene recognition usually relies on more advanced semantic understanding, the server may choose a multimodal model that can combine image and text information to understand the scene. However, in this example, the scene recognition is relatively direct (i.e., sunrise), so there may be no need to call the model additionally, but the object detection results and text descriptions can be directly used for filtering. For the image retrieval task, the server selects the image vectorization model to convert the user's uploaded image into a vector representation for retrieval in the vector database. Selecting a retrieval interface: For object detection and scene recognition tasks, since they do not directly involve external search or database retrieval, no specific retrieval interface is required. For image retrieval tasks, the server selects a vector retrieval interface to retrieve images similar to the user-uploaded image vectors in a pre-established vector database. In addition, in order to sort and optimize the retrieval results, the server may also use a sorting interface to score and sort the retrieved images.

[0087] The server performs the search task according to the selected model and interface. First, the open vocabulary object detection model is used to identify the peaks in the image and extract relevant features. Then, the results are preliminarily filtered in combination with the text description "Sunrise Peak". Next, the image uploaded by the user is converted into a vector representation using the image vectorization model, and similar images are retrieved from the vector database through the vector retrieval interface. Finally, the server uses the sorting interface to score and sort the retrieved images to obtain the final search results. The server successfully retrieved the sunrise peak images similar to the user's uploaded images from the image library and sorted them according to relevance. These search results will be returned to the user to meet the user's search needs.

[0088] In an embodiment of the present invention, the image search agent calls the target models corresponding to the at least one task to be performed to execute the at least one task to be performed and obtains the corresponding task execution result, which can be implemented through the following examples.

[0089] Based on the image search agent, the target model corresponding to the at least one task to be executed is called to execute the at least one task to be executed, and a corresponding execution result of the task to be determined is obtained;

[0090] Observe the execution result of the pending task through the image search agent and obtain the corresponding observation result;

[0091] When the observation result indicates that the pending task execution result is normal, obtaining a summary result based on the pending task execution result, and using the summary result as the task execution result;

[0092] When the observation result indicates that the execution result of the pending task is abnormal, repeatedly refer to the execution result of the pending task and call the target model corresponding to the at least one pending task to execute the at least one pending task until the observation result indicates that the execution result of the pending task is normal.

[0093] In an embodiment of the present invention, exemplarily, after receiving the user's search request, the server determines multiple tasks to be performed after task splitting. Next, the server will call the corresponding target model based on the image search agent to perform these tasks, and preliminarily obtain the execution results of the pending tasks. Suppose the user uploads a picture containing a mountain and a sunrise, and hopes to find similar pictures. After task splitting, the server determines the following tasks to be performed: use the target detection model to identify the mountain in the picture, and use the picture vectorization model to convert the picture into a vector. The server first calls the target detection model to perform the task of identifying the mountain. After the model runs, the location and feature information of the mountain are returned, which is the first result of the execution of the pending task. Then, the server calls the picture vectorization model to convert the picture into a vector representation, and obtains the vector information of the picture, which is the second result of the execution of the pending task.

[0094] After obtaining the pending task execution results, the server will not immediately output them as the final results, but will observe and analyze these results through the image search agent to determine whether they are normal or valid. For the output results of the target detection model, the image search agent will check whether the returned mountain peak position and feature information are reasonable. For example, it will confirm whether the detected mountain peaks actually exist in the image and whether the position annotation is accurate. If everything is normal, the observation result will indicate that the pending task execution results are normal; if any problems are found (such as no peaks detected or incorrect position annotations), the observation result will indicate anomalies. For the output results of the image vectorization model, the image search agent will check whether the generated vector is as expected. For example, it will verify whether the vector is valid (non-zero vector) and whether it contains enough information to distinguish different images. If the vector is valid and informative, the observation result will indicate normality; otherwise, it will indicate anomalies. If the image search agent observes that the pending task execution results are normal, the server will generate a summary result based on these results and output it as the final task execution result. In this example, the server combines the mountain peak position and feature information identified by the target detection model and the vector information generated by the image vectorization model to form the final search result description. If the image search agent observes abnormal results of pending tasks, the server will not directly output these results, but will repeat the related tasks. In this example, if the object detection model fails to correctly identify the peaks or the vectors generated by the image vectorization model are invalid, the server will call these models again to perform the task. This process may be repeated many times until the image search agent observes normal results of pending tasks. Each time a task is re-executed, the server will refer to the previous pending task execution results to optimize the model input or parameter settings to improve the execution effect of the task.

[0095] In the embodiment of the present invention, the preset offline image library is constructed in the following manner.

[0096] Get the image dataset of the preset business type;

[0097] Processing the image data set based on the image storage agent to obtain a text analysis structure and vector analysis results for each image;

[0098] A corresponding index and a persistent storage structure are established according to the text analysis structure and the vector analysis result to construct the preset offline image library.

[0099] In an embodiment of the present invention, illustratively, the first step for the server to start building a preset offline image library is to obtain a picture data set related to a specific business type. These picture data sets will serve as the basic data of the offline image library. Assume that the server is building an offline image library focused on tourist attractions. The server will first collect pictures related to tourist attractions from various sources (such as travel websites, picture libraries, user uploads, etc.) to form a huge picture data set. This data set contains pictures of various tourist attractions (such as famous mountains, rivers, historical sites, etc.).

[0100] The server uses a pre-trained image storage agent to process the collected image dataset. This agent can analyze the content of the image, extract useful information, and generate a text analysis structure and vector analysis results for each image. The server inputs the image dataset into the image storage agent. The agent first calls the target detection model to identify key objects (such as mountains, ancient buildings, lakes, etc.) and scenes (such as sunrise, sunset, four seasons, etc.) in the image. Then, it may also call the OCR model to identify text information in the image (such as scenic spot names, introductions, etc.). In addition, the agent will also call the image vectorization model to convert each image into a vector representation for subsequent similarity comparison and retrieval. After these processing steps, each image will obtain a text analysis structure (containing the identified objects, scenes, and text information) and a vector analysis result (vector representation of the image).

[0101] After obtaining the text analysis structure and vector analysis results of each image, the server needs to create an index and persistent storage structure for these results so that related images can be quickly retrieved later. The server first selects an appropriate storage technology to store the text analysis structure and vector analysis results of the image. For the text analysis structure, the server can use text search engines such as Elasticsearch to create indexes so that text information can be quickly retrieved. For the vector analysis results, the server can choose to use a dedicated vector database (such as Milvus, Faiss, etc.) to store vector data and support efficient vector similarity calculation and retrieval. When creating the index and storage structure, the server assigns a unique identifier (such as ID) to each image and associates this identifier with the corresponding text analysis structure and vector analysis results. In this way, in the subsequent retrieval process, the server can quickly locate the relevant image and its analysis results through this identifier. After the above steps, the server successfully built a preset offline image library containing rich tourist attraction images, and supports efficient retrieval operations through text and vector information.

[0102] The embodiments of the present invention also include the following implementation modes.

[0103] When the target image cannot be extracted from the preset offline image library based on the task execution result,

[0104] An external search engine is called to search for an image corresponding to the task execution result as the target image through the external search engine.

[0105] In an embodiment of the present invention, illustratively, the server has executed a series of tasks based on the user's search request and obtained corresponding task execution results. However, when trying to extract the target image from the preset offline image library, the server finds that there are no pictures in the library that match the task execution results. Suppose a user uploads a picture containing a unique plant and hopes to find similar plant pictures. The server identifies the plant species in the picture through the target detection model and converts the picture into a vector representation through the picture vectorization model. However, when trying to retrieve similar pictures from the preset offline image library, the server finds that there are no pictures of this plant in the library.

[0106] When the preset offline image library cannot meet the user's search needs, the server needs to find other ways to obtain the target image. In this case, the server will call an external search engine to use a wider range of network resources to search for images that match the task execution results. The server calls the API interface of one or more external search engines and sends the task execution results (including text descriptions of plant species and image vector representations) as search parameters to the external search engine. The external search engine will search for similar images in its huge image database based on these parameters.

[0107] The external search engine performs a search operation in its image database according to the search parameters sent by the server, and returns a list of images that match the task execution results. The server selects the image that best meets the user's needs from these images as the target image. The external search engine finds multiple images similar to the plant species in the uploaded image in its image database, and returns them to the server in order of similarity. The server selects several images that best meet the needs as target images based on the sorting results and the user's search intent. These target images will be returned to the user to meet the user's search needs. Through this process, even if the preset offline image library does not have the image the user needs, the server can still find similar images by calling the external search engine and return them to the user as search results. This greatly expands the scope and accuracy of the search and improves the user's search experience.

[0108] In order to more clearly describe the solution provided by the embodiments of the present application, an overall implementation method is provided below.

[0109] The embodiment of the present invention includes two core modules:

[0110] Offline storage module: extracts the features of the image and stores them persistently. It is used by the online query module to respond to user queries. The information that users can retrieve includes text and vectors.

[0111] Online query module: retrieve images that meet user needs from the library. When the search results in the library are empty, it supports obtaining results from external APIs, such as Baidu, Google, etc.

[0112] Online query module:

[0113] Please refer to Figure 2 , Figure 2 A schematic diagram of the overall framework of an image search system provided by an embodiment of the present invention.

[0114] 1. Overall Architecture

[0115] User input:

[0116] The present invention supports three modes of user input:

[0117] Plain text input: For example, "Help me search for photos of a certain celebrity."

[0118] Pure image input: For example, directly inputting an image without entering any text information.

[0119] Image and text input: For example, text input "Help me search for a picture with the same posture as the person in the picture below", image input.

[0120] User intention judgment:

[0121] Some user input is unclear, or the user's needs have multiple requirements, and the user will enter them one by one into the text box. At this time, it is necessary to interact with the user and wait for the user to fully express his intention. For example, if the user enters "Search for a picture similar to the one below", and the user has not entered the picture at this time, then there is no need to go to the search logic at this time. It is necessary to use LLM to give the user some feedback and provide better user interaction. For example, in this case, return the following information to the user: "OK, I will search for a similar picture for you, please provide the picture you need to search for", and wait until the user's intention is clear and the necessary data is obtained before entering the search logic.

[0122] Image Search Agent - Workflow Description:

[0123] This module is the core module of the present invention. Here we will first explain its function, and the implementation principle will be elaborated in detail in the following chapters. For example, if a user wants to search for Figure 3 Similar photos. According to human logic, the following thoughts and steps will be generated

[0124] Think: What is in this picture? I need to perform the following steps. Step 1: I need to use the object detection model to identify what is in the photo. Step 2: Then execute the next plan based on the result of the object detection. Step 3: If the object detection returns some results, then search the library. Step 4: If the object detection model does not detect any objects, then I need to execute the captions model to see if there is any other useful information.

[0125] Execution: Calling the target detection model

[0126] Observation: The result returned by the object detection model is a table, a desk lamp, three books, a stool, five pens, a potted plant, and a schoolbag.

[0127] Think: I have detected "a table, a desk lamp, three books, a stool, five pens, a potted plant, and a schoolbag". What is the positional relationship of these objects? I need to perform the following tasks. Step 1: Call the scene graph to generate the model

[0128] Execution: Call the scene graph to generate the model

[0129] Observation: The scene graph generation model returns the result as “(lamp, on, table), (book, on, table), (pen, on, table), (plant, on, book), (schoolbag, hang, table), (stool, before, table)”

[0130] Thinking: I have obtained the objects in this photo and know their positional relationship. I need to perform the following tasks. Step 1: Call the image vectorization model to obtain some candidate images. Step 2: Compare the candidate images with the query image information in turn. Step 3: Sort the results.

[0131] Execution: Calling the image vectorization model

[0132] Observation: Some candidate images were obtained

[0133] Think about it: We have obtained candidate images and we already know the information in the images that the user is looking for. Now we compare the results, score each result, and sort the results based on the scores. We return the results to the user.

[0134] The above steps roughly reflect the workflow of the image search agent, but this workflow is given based on the example given above. In fact, different user inputs will produce different workflows, and this workflow is not designed in advance, but is the result of continuous thinking by the image search agent based on user input and the return results of the model.

[0135] Please refer to Figure 4 , Figure 4 A schematic diagram of the workflow of an image search agent provided in an embodiment of the present invention.

[0136] 2. Image Search Agent

[0137] The image search agent is the core of the present invention. The workflow of the image search agent has been demonstrated above with specific examples. The principle of the image search agent is explained below. With the help of the general task capabilities of LLM, the image search agent plays a role similar to the human "brain" in the present invention, simulating the human problem-solving ideas. In the process of solving complex problems, humans often think and execute step by step, and will adjust the next step of work ideas at any time according to the results of the previous step.

[0138] The image search agent simulates the above human thinking process and completes the image search task through the following three modules' working cycles. The multiple cycles generally follow the process of first analyzing the user's query image and then calling the retrieval API.

[0139] Task splitting (thinking)-workflow description:

[0140] This module acts as the brain. Since the image search agent may go through multiple cycles to complete the user's search needs, this module is divided into two cases: the first call and the non-first call. (Note: In the actual working process, there is no distinction between the first and the non-first cases. This is just for the convenience of explanation.) Here we first explain the workflow of the model, and the implementation principle will be explained in the following chapters.

[0141] First call:

[0142] First, based on the user's input text (user intent) and image, think roughly about what needs to be done to solve this problem. For example, if the user inputs the text "Help me search for photos of this celebrity in the photo" or "Help me search for photos of the behavior in the photo", the search ideas are different for these two inputs. For the former, you need to call face detection first, then face recognition, and then search the library for photos of the recognized person; for the latter, you may need to call the scene graph generation model or the behavior recognition model (depending on which models are provided), and then search the library after recognizing the behavior.

[0143] Not the first call:

[0144] At this point, we have obtained some model call results. Here we need to plan what to do next based on the model results. For example, in the example of "Help me search for photos of this celebrity in the photo", the face detection model call here may not detect a face, or may detect multiple celebrities. Then we need to confirm the information with the user. If there is no face, we will inform the user that no face was detected and ask him to upload the picture again. If multiple celebrities were identified, we need to ask the user which one he wants to search for or whether he wants to search for all of them.

[0145] In short, this module needs to formulate the next round of execution plan based on user input or the execution results of the previous round of models. When it is determined that the results can be returned to the user, the results are returned to the user.

[0146] Call the model or retrieve (execute):

[0147] The previous step has listed the execution plan (the model to be called or whether to start the search). This module is responsible for calling the corresponding model and search API according to the execution plan.

[0148] Observe the execution results (observation):

[0149] This module is responsible for summarizing the execution results of the previous step. When the model is called abnormally or the model returns empty, the observed results are also reported to the task splitting module.

[0150] Please refer to Figure 5 , Figure 5 A schematic diagram of the steps of task splitting provided in an embodiment of the present invention.

[0151] 3. Task Splitting-Implementation Principle

[0152] The task splitting module is the core of the image search agent. Task splitting is achieved with the help of the general task capability of LLM. The input of LLM has two parts:

[0153] (1) User input or the results of the previous round of model execution (not the first execution)

[0154] (2) Prompt: Prompt is used to prompt LLM to perform tasks.

[0155] In the present invention, the Prompt of this module includes three parts.

[0156] (1) Prompt used to describe the task

[0157] (2) Visual model list

[0158] (3) Example (few-shot learning)

[0159] 1. Prompt to describe the task

[0160] Used to describe the tasks to be performed, as well as some precautions. Examples are as follows:

[0161] "Suppose you are an image search agent and you need to complete an image search task. You will be provided with a list of visual models to help you complete the task. The completion of this task may require multiple loop steps. A complete loop step includes the following three steps: 1. Based on user input or the results of the previous loop, the list of available models outputs a search plan. 2. Based on the search plan, call related models. 3. Summarize the results of model calls. Based on the results of the previous loop, you need to think again and give an execution plan again. Repeat the cycle to directly complete the search task.

[0162] The list of selectable visual models is provided with the model ID and the detailed description of the model, and you will select the model based on the detailed description of the model.

[0163] The retrieval API is divided into two categories, one for recalling images through text, and the other for recalling images through vectors.

[0164] You need to make use of these models as much as possible to obtain as much information as possible from the image. The input of some models may need to depend on the output of other models, and you need to sort out these relationships.

[0165] To help you complete the task better, you will be provided with 3 examples. "

[0166] Note: The prompt above is just an example. For different LLMs, continuous debugging is required to achieve the desired effect.

[0167] 2. Visual model list and retrieval API

[0168] It is used to inform LLM of the visual models and retrieval APIs that can be called. The following model list is just an example. In actual use, you can add or delete models according to your business. Due to space constraints, the detailed description of the model is omitted below, and only a brief description is given. In actual application, a detailed description of the model is required, which is the key for LLM to understand the model.

[0169] List of visual models

[0170] Captions model:

[0171] Model No.: 001

[0172] Model detailed description: input image, detailed description of output image, omitted...

[0173] Image vectorization model:

[0174] Model No.: 002

[0175] Model description: Input image, output image semantic vector, omitted...

[0176] Open Vocabulary Object Detection Model:

[0177] Model No.: 003

[0178] Model description: Input a picture, output the target category and location in the picture, omitted...

[0179] Face recognition model:

[0180] Model No.: 004

[0181] Model description: Input a picture, output the name of the face in the picture, omitted...

[0182] Animal Identification Model:

[0183] Model No.: 005

[0184] Model description: Input a picture and output the detailed species of the animal in the picture, omitted...

[0185] Flower, grass and tree recognition model:

[0186] Model No.: 006

[0187] Model description: Input a picture and output the detailed species of flowers, plants and trees in the picture, omitted...

[0188] OCR Model:

[0189] Model No.: 007

[0190] Model description: Input a picture, output the location and recognition results of the text in the picture, omitted...

[0191] Human key point retrieval model:

[0192] Model No.: 008

[0193] Model description: Input a picture and output the key points of the human body in the picture, omitted...

[0194] 3. Retrieval API

[0195] External Image Search API:

[0196] API Number: 101

[0197] API detailed description: Input text, output image list, omit...

[0198] Text Retrieval API:

[0199] API Number: 102

[0200] API Detailed Description: Input text, output image list. The offline storage module has expressed the image in text and persisted the text, so the online query module can use the text to match the image most relevant to the text.

[0201] Vector retrieval API:

[0202] API Number: 103

[0203] API Detailed Description: Input vector, output image list. The offline storage module has expressed the image as a vector and stored it persistently, so the online query module can use the vector to match the image most relevant to the query image.

[0204] Sorting API:

[0205] API Number: 104

[0206] API detailed description: Input the image and the similarity score with the user's requirements, and sort the results.

[0207] Example

[0208] Examples are used to help LLM understand the task, and LLM can learn the idea of ​​splitting the task from the examples.

[0209] “query: Give me more images of the following billboards

[0210] image_url: http: / / xxx / xxx / a.jpg

[0211] think: step1: first call the target detection model to detect the location of the billboard; step2: call the OCR model to recognize the text on the billboard; step3: call the retrieval API based on the text content on the billboard

[0212] execution:

[003]

[0213] observation: {"result": ["board":[0.1, 0.1, 0.2,0.2]]}

[0214] think: The billboard location has been detected. step1: Call the OCR model to recognize the billboard text; step2: Call the search API based on the billboard text content

[0215] execution:

[007]

[0216] observation: {"result": ["Lanzhou Ramen"]}

[0217] think: The billboard location has been detected and the billboard content is known. step1: Call the retrieval API based on the billboard text content

[0218] execution:

[102]

[0219] observation: {"result": ["100.jpg","121.jpg","250.jpg"]}

[0220] think: Complete the search task and return the search results: {"result": ["100.jpg","121.jpg","250.jpg"]}

[0221] ”

[0222] Please refer to Figure 6 , Figure 6 A schematic diagram of the overall framework of the offline warehousing module provided in an embodiment of the present invention.

[0223] Offline storage module:

[0224] The online query module supports searching the self-built image library. When the self-built image library cannot find the content, it will search for the image through an external search engine and return it to the user. The process of entering the self-built image library is shown in the figure. The detailed process is as follows

[0225] Image dataset: a pre-built image dataset for your own business

[0226] Image storage agent: Get as much information as possible about the image. Because the analysis of the image may be divided into different granularities, it is designed as an agent here. For example, when there is a dog in the image, the animal recognition model needs to be called. When there is a person in the image, the face recognition model needs to be called. The image storage agent will autonomously select the next model that can be called based on the call result of the previous model, if any.

[0227] Image analysis results: Summarize and merge the execution results of the image storage agent. Ultimately, one image will produce two types of results.

[0228] Text: For example, objects, locations, scene graphs, behaviors, detailed descriptions, etc. in the image. The online query model can quickly locate these images based on these text tags.

[0229] Vector: Image vectorization can meet the requirements of generalization and speed of retrieval. For example, if there are no photos of apples in the self-built pictures, but there are photos of oranges, then when the user searches for pictures of apples, it is also a good choice to return pictures of oranges to the user.

[0230] Storage: The analysis results of the image are text and vector. Both need to be indexed and persistently stored to facilitate retrieval by the online query module. For text, you can choose Elasticsearch storage, and for image vectors, you can choose vector database storage, such as Milvus, Faiss, etc.

[0231] In summary, the image search method provided by the embodiment of the present invention allows the user to interact with the system for multiple rounds to further refine or limit the search results. When the user is not satisfied with the initial search results, the system can combine the previous search results and the current user's intention to provide more accurate search results, thereby more effectively meeting the user's search needs. The method has a high degree of autonomy and can autonomously select and call a suitable retrieval model according to the user's intention. For complex search needs or situations that depend on the results of other models, the system can autonomously disassemble the task and call the appropriate model in multiple rounds until the user's search task is completed. This autonomy greatly improves the efficiency and accuracy of the search. The method supports the pluggable design of the model. When a new model is added, the system only needs to simply provide the description, input and output format of the model, and the new model can be easily incorporated into the search process. When a new model is added, by adding a new prompt example, the search path of the existing image search agent can be improved to make it more adaptable to new search scenarios and needs. During the model call process, if an exception or error occurs, the system can observe the exception and autonomously think about alternative models for calling to ensure that the search process is not interrupted, thereby improving the stability and reliability of the system. By introducing the design of intelligent agents, this method can call multiple models during the image search process and design the calling order according to the specific situation. This intelligence makes the search results more accurate and comprehensive, which is better than the image search method that only uses a single model. This method supports users' search needs in the form of free dialogue, providing a more natural and convenient way of interaction. Users can express their search intentions through simple text or picture input, and the system can respond quickly and provide search results that meet their needs, thereby greatly improving the user experience.

[0232] Please refer to Figure 7 , Figure 7 An image search device 110 provided in an embodiment of the present invention includes:

[0233] The acquisition module 1101 is used to acquire at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement;

[0234] The execution module 1102 is used to input the multimodal search requirement instruction into a pre-trained image search agent when the at least one multimodal search requirement instruction represents a complete user intention; use the image search agent to split the multimodal search requirement instruction into tasks to obtain at least one task to be executed corresponding to the multimodal search requirement instruction; based on the image search agent, call the target model corresponding to the at least one task to be executed to execute the at least one task to be executed, and obtain the corresponding task execution result;

[0235] The output module 1103 is used to extract a target image from a preset offline image library as an image search result corresponding to the multimodal search requirement instruction based on the task execution result.

[0236] It should be noted that the implementation principle of the aforementioned image search device 110 can refer to the implementation principle of the aforementioned image search method, and will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the image search device 110 can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The function of the above image search device 110. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0237] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0238] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned image search device 110. Figure 8 As shown, Figure 8The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes an image search device 110, a memory 111, a processor 112 and a communication unit 113.

[0239] To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be achieved through one or more communication buses or signal lines. The image search device 110 includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the image search device 110 stored in the memory 111, such as the software function modules and computer programs included in the image search device 110.

[0240] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.

Claims

1. An image search method, characterized in that: include: Acquire at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement; In a case where the at least one multimodal search requirement instruction represents a complete user intention, inputting the multimodal search requirement instruction into a pre-trained image search agent; Using the image search agent to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction; Based on the image search agent, the target model corresponding to the at least one task to be performed is called to perform the at least one task to be performed, and a corresponding task execution result is obtained; Based on the task execution result, extracting a target image from a preset offline image library as an image search result corresponding to the multimodal search requirement instruction; The method further comprises: Receiving a user's feedback multimodal search demand instruction for the image search results; Based on the feedback multimodal search demand instruction, calling the image search agent to perform image retrieval in combination with the image search results to obtain candidate image search results; When receiving a multimodal search demand instruction of a user's feedback for the candidate image search results, calling the image search agent to perform image retrieval in combination with the candidate image search results; The at least one multimodal search requirement instruction includes a first multimodal search requirement instruction and a second multimodal search requirement instruction, and the obtaining of the at least one multimodal search requirement instruction input by the user includes: Obtaining the first multimodal search requirement instruction; Parsing the first multimodal search requirement instruction to obtain a first parsing result; Determining whether the first parsing result constitutes a complete intent requirement; If so, determining that the first multimodal search requirement instruction represents a complete user intention; If not, generating a feedback interaction instruction based on the first multimodal search requirement instruction; Acquire the second multimodal search requirement instruction input by the user in response to the feedback interaction instruction; Parsing the second multimodal search requirement instruction to obtain a second parsing result; Determining whether the first analysis result combined with the second analysis result constitutes a complete intention requirement; If so, determining that the first multimodal search requirement instruction combined with the second multimodal search requirement instruction represents a complete user intention; If not, the step of generating feedback interaction instructions is repeated until a multimodal search requirement instruction representing the complete user intention is obtained; The step of using the image search agent to split the multimodal search requirement instruction into tasks to obtain at least one to-be-executed task corresponding to the multimodal search requirement instruction includes: Calling a pre-trained general task classification large language model to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction; The step of calling the target model corresponding to the at least one task to be executed based on the image search agent to execute the at least one task to be executed and obtaining the corresponding task execution result includes: Obtaining a task prompt word corresponding to the at least one task to be performed; Determine, according to the task prompt word, a model list and a retrieval interface corresponding to the at least one task to be performed, and call a target model corresponding to the at least one task to be performed from the model list; Execute the at least one task to be executed in combination with the retrieval interface and the target model to obtain a corresponding task execution result; The model list includes a picture-text multimodal model, a picture vectorization model, an open vocabulary target detection model, a target object recognition model, a text extraction model, and a key point recognition model; The retrieval interface includes an external image search interface, a text retrieval interface, a vector retrieval interface, and a sorting interface; The step of calling the target model corresponding to the at least one task to be executed based on the image search agent to execute the at least one task to be executed and obtaining the corresponding task execution result includes: Based on the image search agent, the target model corresponding to the at least one task to be executed is called to execute the at least one task to be executed, and a corresponding execution result of the task to be determined is obtained; Observe the execution result of the pending task through the image search agent and obtain the corresponding observation result; When the observation result indicates that the pending task execution result is normal, obtaining a summary result based on the pending task execution result, and using the summary result as the task execution result; When the observation result indicates that the execution result of the pending task is abnormal, repeatedly executing the step of referring to the execution result of the pending task and calling the target model corresponding to the at least one pending task to execute the at least one pending task, until the observation result indicates that the execution result of the pending task is normal; The preset offline image library is constructed in the following way: Get the image dataset of the preset business type; Processing the image data set based on the image storage agent to obtain a text analysis structure and vector analysis results for each image; Establishing a corresponding index and a persistent storage structure according to the text analysis structure and the vector analysis result, and constructing the preset offline image library; The method further comprises: When the target image cannot be extracted from the preset offline image library based on the task execution result, An external search engine is called to search for an image corresponding to the task execution result as the target image through the external search engine.

2. An image search device, characterized in that: include: An acquisition module, used to acquire at least one multimodal search requirement instruction input by a user, wherein the multimodal search requirement instruction includes at least one of a text requirement and an image requirement; An execution module is used to input the multimodal search requirement instruction into a pre-trained image search agent when the at least one multimodal search requirement instruction represents a complete user intention; use the image search agent to split the multimodal search requirement instruction into tasks to obtain at least one task to be executed corresponding to the multimodal search requirement instruction; based on the image search agent, call the target model corresponding to the at least one task to be executed to execute the at least one task to be executed, and obtain the corresponding task execution result; An output module, used to extract a target image from a preset offline image library as an image search result corresponding to the multimodal search requirement instruction based on the task execution result; The execution module is also used for: Receive a user's feedback multimodal search demand instruction for the image search results; based on the feedback multimodal search demand instruction, call the image search agent to perform image retrieval in combination with the image search results to obtain candidate image search results; when receiving the user's feedback multimodal search demand instruction for the candidate image search results, call the image search agent to perform image retrieval in combination with the candidate image search results; The at least one multimodal search requirement instruction includes a first multimodal search requirement instruction and a second multimodal search requirement instruction, and the acquisition module is specifically used to: acquire the first multimodal search requirement instruction; parse the first multimodal search requirement instruction to obtain a first parsing result; and determine whether the first parsing result constitutes a complete intention requirement; If so, determining that the first multimodal search requirement instruction represents a complete user intention; If not, generating a feedback interaction instruction based on the first multimodal search requirement instruction; obtaining the second multimodal search requirement instruction input by the user in response to the feedback interaction instruction; parsing the second multimodal search requirement instruction to obtain a second parsing result; determining whether the first parsing result combined with the second parsing result constitutes a complete intention requirement; If so, determining that the first multimodal search requirement instruction combined with the second multimodal search requirement instruction represents the complete user intention; if not, repeating the step of generating feedback interaction instructions until a multimodal search requirement instruction representing the complete user intention is obtained; The execution module is specifically used for: Calling a pre-trained general task classification large language model to perform task splitting on the multimodal search demand instruction to obtain at least one to-be-executed task corresponding to the multimodal search demand instruction; Obtaining a task prompt word corresponding to the at least one task to be performed; Determine, according to the task prompt word, a model list and a retrieval interface corresponding to the at least one task to be performed, and call a target model corresponding to the at least one task to be performed from the model list; Execute the at least one task to be executed in combination with the retrieval interface and the target model to obtain a corresponding task execution result; The model list includes a picture-text multimodal model, a picture vectorization model, an open vocabulary target detection model, a target object recognition model, a text extraction model, and a key point recognition model; The retrieval interface includes an external image search interface, a text retrieval interface, a vector retrieval interface, and a sorting interface; Based on the image search agent, the target model corresponding to the at least one task to be executed is called to execute the at least one task to be executed, and a corresponding execution result of the task to be determined is obtained; Observe the execution result of the pending task through the image search agent and obtain the corresponding observation result; When the observation result indicates that the pending task execution result is normal, obtaining a summary result based on the pending task execution result, and using the summary result as the task execution result; When the observation result indicates that the execution result of the pending task is abnormal, repeatedly executing the step of referring to the execution result of the pending task and calling the target model corresponding to the at least one pending task to execute the at least one pending task, until the observation result indicates that the execution result of the pending task is normal; The preset offline image library is constructed in the following way: Get the image dataset of the preset business type; Processing the image data set based on the image storage agent to obtain the text analysis structure and vector analysis results of each image; establishing corresponding indexes and persistent storage structures according to the text analysis structure and the vector analysis results to construct the preset offline image library; The execution module is also used for: When the target image cannot be extracted from the preset offline image library based on the task execution result, an external search engine is called to search for an image corresponding to the task execution result as the target image through the external search engine.

Images