Image searching method and device, intelligent agent, electronic equipment and storage medium

Abstract

The invention provides an image search method and device, an intelligent agent, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models, image search and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. The method comprises the steps that according to input information used for image search, multi-modal search information is determined, the input information comprises first text information and / or a first reference image, and the search information comprises second text information and a second reference image; performing text analysis on the second text information and first description information used for describing a second reference image by utilizing a text analysis large model to generate at least one piece of second description information; and determining at least one target image according to the at least one piece of second description information, each target image being determined according to the at least one piece of second description information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to computer vision, deep learning, large models, image search and other technical fields, and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Specifically, it relates to an image search method, device, intelligent body, electronic device and storage medium. Background Art

[0002] With the continuous development of artificial intelligence technology, big model technology has also been applied to many fields. For example, image search can be realized with the help of big models.

[0003] However, when performing image search based on large models, image search with multiple modal inputs corresponds to multiple processing methods, the system complexity and maintenance cost are high, and it is also difficult to implement image search in complex scenarios, such as flexibly switching between multiple modal inputs for image search. Summary of the invention

[0004] The present disclosure provides an image search method, device, intelligent agent, electronic device and storage medium.

[0005] According to one aspect of the present disclosure, there is provided an image search method, comprising: determining multimodal search information based on input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image; using a text analysis large model, performing text analysis on the second text information and first description information for describing the second reference image to generate at least one second description information; and determining at least one target image based on the at least one second description information, wherein each target image is determined based on the at least one second description information.

[0006] According to another aspect of the present disclosure, an image search device is provided, including: a first determination module, used to determine multimodal search information based on input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image; a generation module, used to perform text analysis on the second text information and first description information for describing the second reference image using a text analysis large model, and generate at least one second description information; and a second determination module, used to determine at least one target image based on the at least one second description information, wherein each target image is determined based on the at least one second description information.

[0007] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the method provided by the embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture to which the image search method and device according to an embodiment of the present disclosure can be applied is schematically shown;

[0014] Figure 2 A flowchart of an image search method according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A schematic diagram of a scenario for determining search information according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4 A schematic diagram of a scenario for obtaining input information according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 5 A schematic diagram of a scene for determining at least one target image according to an embodiment of the present disclosure is schematically shown;

[0018] Fig. 6A A schematic diagram schematically shows a scenario of generating second description information according to an embodiment of the present disclosure;

[0019] Figure 6B A schematic diagram schematically shows a scenario of generating second description information according to another embodiment of the present disclosure;

[0020] Figure 7A schematic diagram schematically shows a scenario of determining a target image from a search image library according to a specific embodiment of the present disclosure;

[0021] Figure 8 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown;

[0022] Fig. 9 A block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown;

[0023] Fig.10 A block diagram schematically shows a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0024] Fig.11 A block diagram of an electronic device 1100 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Currently, image search tasks based on large models include text image search tasks, composed image retrieval (CIR) tasks, and conversational image search (Chat-based Image Retrieval, Chat-IR) tasks, and their inputs are: text input in pure language mode, multi-modal input of reference image + text instructions, and text input obtained by combining multiple rounds of conversations.

[0027] On the one hand, due to the variety of input forms required for different tasks, traditional single-modality image search methods need to design different model architectures and optimization strategies for each task, which increases the complexity and computational cost of the system. In addition, this separate design increases the complexity of the system and requires the system to adapt between different tasks, which increases additional development and maintenance costs.

[0028] On the other hand, as user needs diversify, the use scenarios of image search are often no longer limited to a single task mode. In complex application scenarios, users' image search needs may be cross-task and dynamically changing. For example, at the beginning, users may only want to perform simple image searches through text, but as the interaction deepens, they may require the system to combine a reference image or use dialogue to further optimize the search results. Existing image search methods are difficult to flexibly respond to such cross-task, complex and dynamic needs because they separate various tasks in design, which leads to a poor image search experience in complex scenarios.

[0029] In order to at least partially solve the above technical problems, an embodiment of the present disclosure provides an image search method, comprising: determining multimodal search information based on input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image. Using a large text analysis model, text analysis is performed on the second text information and the first description information used to describe the second reference image to generate at least one second description information. Based on the at least one second description information, at least one target image is determined, wherein each target image is determined based on the at least one second description information. Through the embodiments of the present disclosure, the technical problems of high system complexity and maintenance cost and poor search experience can be at least partially solved, and the technical effect of reducing system complexity and maintenance cost and improving search experience can be achieved.

[0030] Figure 1 An exemplary system architecture to which the image search method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0031] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the image search method and apparatus can be applied may include a terminal device, but the terminal device may implement the method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0032] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0033] The user may use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software, etc. (only as examples).

[0034] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0035] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for the content browsed by the user using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0036] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0037] It should be noted that the image search method provided in the embodiment of the present disclosure can generally be executed by the first terminal device 101, the second terminal device 102, and the third terminal device 103. Accordingly, the image search device provided in the embodiment of the present disclosure can also be set in the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0038] Alternatively, the image search method provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the image search device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The image search method provided in the embodiment of the present disclosure may also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the image search device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0039] For example, a user inputs input information for image search through the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the first terminal device 101, the second terminal device 102, and the third terminal device 103 determine multimodal search information based on the input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image; using a text analysis large model, text analysis is performed on the second text information and the first description information for describing the second reference image to generate at least one second description information; and based on the at least one second description information, at least one target image is determined, wherein each target image is determined based on the at least one second description information.

[0040] Alternatively, input information for image search is sent to the server 105 through the first terminal device 101, the second terminal device 102, and the third terminal device 103, and the server 105 performs the above-mentioned image search method to determine and return at least one searched target image to the first terminal device 101, the second terminal device 102, and the third terminal device 103.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0042] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0043] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0044] Figure 2The flowchart of the image search method according to the embodiment of the present disclosure is schematically shown.

[0045] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0046] In operation S210, multimodal search information is determined according to input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image.

[0047] The input information may be a single-modal input form, such as first text information in a single language mode, or a first reference image in a single visual mode. Alternatively, the input information may also be multi-modal input information, such as first text information and a first reference image. For example, the input information may be a pure text input for a text image search task, an input of a reference image and a text instruction for a combined image search task, and a combined text input of multiple rounds of dialogue for a dialogue image search task.

[0048] For example, the input information includes first text information, such as "a girl is playing with a white cat"; or, the input information includes a first reference image, such as an image representing "a girl is playing with a black cat"; or, the input information includes the first text information and the first reference image, the first reference image is an image representing "a girl is playing with a black cat", and the first text information is "change the black cat to a white cat".

[0049] The search information is in a multimodal form and includes the second text information and the second reference image. It is understandable that the search information may be information obtained by standardizing input information in multiple input forms, and the standardization may be the unification of modality, form, size, etc. For example, for modality unification, a single modality or multimodal input form is processed into a multimodal form including the second text information and the second reference image to obtain multimodal search information.

[0050] For example, the first text information may be directly used as the second text information, or the first text information may be processed to obtain the second text information, or the second text information may be a predetermined type of second text information. For example, the first text information may be modified, converted into sentences, segmented, or information extracted to obtain the second text information.

[0051] For another example, the first reference image can be directly used as the second reference image, or the first reference image can be processed to obtain the second reference image, or a predetermined type of second reference image can be used. For example, the first reference image can be processed by cropping, rotating, color correction, etc. to obtain the second reference image.

[0052] In operation S220, a text analysis is performed on the second text information and the first description information for describing the second reference image using a text analysis model to generate at least one second description information.

[0053] The first description information of the second reference image is used to describe the content of the second reference image. For example, the first description information is used to describe the information with language meaning, such as elements, attributes, spatial relationships, etc. in the second reference image. Elements can be objects, such as people, animals, plants, objects, etc., attributes can be features of elements, which can distinguish elements, such as color, shape, size, etc., and spatial relationships can be understood as relative positional relationships between elements.

[0054] The text analysis large model may be a large language model (LLM) for processing language modalities. In this embodiment, the text analysis large model may be a pre-trained large language model.

[0055] By using the large text analysis model, the first description information and the second text information can be semantically analyzed from the text perspective to determine the second description information by integrating the semantics of the two parts. The second description information is also the description information of the image that meets the user's image search needs.

[0056] In one embodiment, the text analysis model can be used to perform a text analysis on the second text information and the first description information to obtain a second description information. Alternatively, the text analysis model can be used to perform multiple text analyses on the same information to obtain second description information that meets the user's image search requirements and has multiple expressions.

[0057] In another embodiment, the text analysis model may be used to perform a text analysis on the second text information and the first description information, but at least one second description information of at least one form may be generated. Alternatively, at least one second description information of at least one semantic granularity may be generated.

[0058] For example, taking the first reference image as an image representing "a girl playing with a black cat" and the first text information as "change the black cat to a white cat", the second description information output by the text analysis model can be "a girl playing with a white cat".

[0059] In operation S230, at least one target image is determined according to the at least one second description information, wherein each target image is determined according to the at least one second description information.

[0060] The at least one second description information may be regarded as description information of an image that meets the image search requirement of the user. Therefore, the target image may be searched for by combining all or part of the at least one second description information.

[0061] For example, a candidate image is searched for according to each second description information, and at least one target image is determined according to the number of second description information hit by the searched candidate images.

[0062] Alternatively, the similarity between each second description information and the candidate image is calculated, and the similarities between one or more second description information and the same candidate image are combined to determine whether to use the candidate image as the target image.

[0063] In the embodiments of the present disclosure, since input information in various forms is unified into multimodal search information, this embodiment can perform image search on input information of various tasks, cross-task scenarios, and modal dynamic changes, without the need to design multiple processing flows in a targeted manner, simplifying the complexity of the system, and being able to flexibly respond to different tasks and meet the needs of users in various image search scenarios. In addition, the second text information and the first description information are semantically analyzed using a large text analysis model to generate at least one second description information, and each target image is determined using at least one second description information, which can make full use of the rich second description information output by the large text analysis model, thereby improving the accuracy of image search.

[0064] Figure 3 A schematic diagram of a scenario for determining search information according to an embodiment of the present disclosure is schematically shown.

[0065] like Figure 3 As shown, embodiment 300 includes operations S301 to S306, which can be used as a specific embodiment of operation S210.

[0066] In operation S301, it is determined whether the input information includes the first text information. After the input information is acquired, whether the input information includes the first text information or not, operation S302 is performed to further determine whether the input information includes the first reference image.

[0067] In operation S302, it is determined whether the input information includes the first reference image. If the input information includes the first text information but does not include the first reference image, operation S303 is performed; if the input information includes both the first text information and the first reference image, operation S304 is performed; if the input information does not include the first text information but includes the first reference image, operation S305 is performed; if the input information does not include both the first text information and the first reference image, operation S306 is performed.

[0068] In operation S303, the first text information is determined as second text information, and a second reference image is acquired, where the second reference image includes a blank reference image.

[0069] The input information only includes the first text information. In this case, the input information is input in a single language mode. The first text information can be used as the second text information and together with a predetermined second reference image, form multi-modal search information.

[0070] In an embodiment, the blank reference image may be an image that does not include any content. For example, the first description information of the blank reference image may be “blank image”.

[0071] In operation S304, the first text information is determined as the second text information, and the first reference image is determined as the second reference image.

[0072] The input information includes the first text information and the first reference image at the same time. In this case, the input information is a multimodal input and can be directly used as search information.

[0073] For example, the input information includes a first reference image: the image is an image representing "a girl playing with a black cat", and the input information also includes a first text information "change the black cat to a white cat". The multimodal search information is: an image representing "a girl playing with a black cat", and "change the black cat to a white cat".

[0074] In operation S305, the first reference image is determined as a second reference image, and second text information is acquired, where the second text information includes blank text information.

[0075] The input information only includes the first reference image. In this case, the input information is input in a single visual mode. The first reference image can be used as the second reference image to form multimodal search information together with the predetermined second text information.

[0076] In one embodiment, the blank text information may be empty information including no text, or may be predetermined text information including a text-free instruction. For example, the blank text information may be “empty”, “text-free instruction”, or the like.

[0077] In operation S306, an error is reported. When the input information does not include the first text information and the first reference image, it can be regarded as an erroneous input and an error is reported.

[0078] In one embodiment, the prompt information of the text analysis model may include two placeholders of language modality and visual modality. The first text information and / or first reference information included in the input information is first filled into the placeholder as the second text information and / or the second reference image; for the placeholder with missing content, the blank reference image and / or blank text information is filled in as the second reference image and / or the second text information.

[0079] In the embodiments of the present disclosure, for input text information in a single language modality or a single visual modality, by filling in a blank second reference image or second text information, the image search task is changed from a single modality input form to a unified multi-modal input without affecting the semantic content, so that the text analysis large model can be used to perform semantic analysis under the same input, thereby achieving compatibility with multiple image search tasks and reducing system complexity and maintenance costs.

[0080] In a specific embodiment, the input information may also be other modality information besides the language modality and the visual modality. The other modality information may be converted into the first text information of the language modality and / or the first reference image of the visual modality, thereby realizing the image search task compatible with other modalities.

[0081] For example, for voice information in audio mode, the voice information can be converted into first text information through a voice-to-text processing algorithm; then, the search information is determined based on the input information.

[0082] In the embodiments of the present disclosure, by converting other modal information input by the user into input information, the image search method of the above-mentioned embodiment is not only applicable to a wider range of usage scenarios, but also does not require system process transformation for the newly added modal information. The processing process of multi-modal search information can be reused, thereby improving the scalability and flexibility of the system.

[0083] According to an embodiment of the present disclosure, for operation S210, the image search method further includes: when the input information includes an indication image library, determining the indication image library as the search image library to determine the target image from the search image library. When the input information does not include the indication image library, determining the reference image library as the search image library.

[0084] For example, the indicated image library may be one of the image libraries included in the system for performing the image search method, and the user may limit the search scope by adding the indicated image library to the input information. If the input information does not include the indicated image library, the system may use a default reference image library.

[0085] For example, in the language-guided image search scenario, the input of the image search is a scoring function ,in, A second text message indicating the language mode, a second reference image representing the visual modality, To search the image library, , such as the search image library can include N candidate images. In the text image search task, Can be , in the combined image search task, Can be , in the conversational image search task, Can be , , represents multiple rounds of dialogue, The text information that makes up a round of dialogue.

[0086] In the embodiments of the present disclosure, by determining whether the input information includes an indication image library, the need to limit the search range can be met, the usage scenarios are expanded, and the user experience is better. In addition, for input without limiting the search range, a predetermined reference image library can also be used for subsequent image searches to meet the user's various image search needs.

[0087] According to an embodiment of the present disclosure, input information is obtained based on at least one of the following methods: determining a first text information and / or a first reference image based on input text and input images in at least one round of conversation; determining a first reference image and / or a first text information based on at least one output image and / or input text in at least one round of conversation, wherein the output image is a target image determined and output based on input information before the output image.

[0088] For ease of understanding, four embodiments are taken as examples below to illustrate input information obtained in a language-guided image search scenario.

[0089] Figure 4 The schematic diagram of the scene of obtaining input information according to the embodiment of the present disclosure is schematically shown. Figure 4 As shown, it includes embodiment 400A, embodiment 400B, embodiment 400C and embodiment 400D.

[0090] In the embodiment 400A, input text 411 is obtained through a round of dialogue, such as "a woman walking on a road covered with fallen leaves and full of sunshine" and the input text 411 is determined as the first text information. In this embodiment, the above-mentioned image search method is used to determine the target image 1 412 according to the input information including the first text information.

[0091] Similarly, an input image can be obtained through a round of dialogue and used as the first reference image in the input information to search for a corresponding target image according to the current input.

[0092] In embodiment 400B, the specific needs of the image search are clarified through two rounds of dialogue, for example, the specific needs of the image search are clarified through input text 421, system output question 422 and input text 423. At this time, the first text information can be obtained according to the combination of input text 421 and input text 423, such as "A woman is walking on the road. There are fallen leaves on the road, and the sun shines on the road." The system output question 422 can be a question generated by the conversational large model to clarify the needs of the road, such as "What kind of road is it?" In this embodiment, the above-mentioned image search method can be used to determine the target image 2 424 according to "A woman is walking on the road. There are fallen leaves on the road, and the sun shines on the road."

[0093] In the embodiment 400C, input information including the first text information and the first reference image can be obtained through the input text 431 and the input image 432 in a round of dialogue. At this time, the first text information is also the input text 431, such as "change the road in the image below into a road covered with fallen leaves and full of sunshine". In this embodiment, the target image 3 433 is determined based on the input information including the first text information and the first reference image.

[0094] In embodiment 400D, in a round of dialogue, the first text information is determined based on the input text 441, such as "a woman walking on a road covered with fallen leaves". The above-mentioned image search method is used to determine the target image based on the input information including the first text information, and output it as a result, such as output image 442. In this embodiment, the user originally only wanted to perform a simple image search through text, but as the interaction deepened, the user initiated a second round of dialogue, hoping to combine a reference image to further optimize the image search results. In the second round of dialogue, the input text 443 can be "based on the output image above, the road should be covered with sunshine." At this time, according to the input text 443, the output image 442 can be used as the first reference image, and the above-mentioned image search method can be used again to determine a new target image 444, such as target image 4, based on the input information including the first text information and the first reference image.

[0095] In the embodiments of the present disclosure, for a variety of image search tasks, cross-task scenarios, and input information with dynamically changing modalities, the input information can be unified into multimodal search information to be applicable to a wider range of scenarios, meet more user image search needs, and improve the user experience.

[0096] According to an embodiment of the present disclosure, for operation S220, the text analysis big model is used to perform the first description generation task, and the text analysis big model is used to perform text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: using the text analysis big model to perform the first description generation task to perform semantic analysis on the second text information and the first description information to generate second description information of multiple semantic granularities; wherein the semantic granularity is related to the number and / or attributes of elements in the second description information, and the elements are extracted from the second text information and / or the first description information.

[0097] The first description generation task includes: understanding the second text information, and generating second description information of multiple semantic granularities that meet the user's image search requirements based on the first description information.

[0098] A single input text of a user may contain multiple semantics, for example, operations on multiple elements, multiple operations on elements, multiple descriptions of elements, etc. Therefore, the text analysis model can be used to perform the first description generation task and generate second description information of multiple semantic granularities, so as to generate clear and rich second description information.

[0099] The second description information of multiple semantic granularities can be understood as multiple second description information including multiple amounts of information, where the amount of information can be the number of elements and / or the number of attributes included. The fewer the number of elements and / or the number of attributes, the smaller the semantic granularity; conversely, the greater the number, the larger the semantic granularity.

[0100] For example, taking three semantic granularities as an example, the first semantic granularity, the second semantic granularity, and the third semantic granularity are Core Elements (CE), Enhanced Details (ED), and Comprehensive Synthesis (CS), respectively. The first semantic granularity may be to include only elements appearing in the second text information without using attributes, where the attributes can be regarded as adjectives, adverbials, attributives, etc. in the second text information. The second semantic granularity may be to include only elements appearing in the second text information, and to use necessary attributes from the second description information. The third semantic granularity may be: to include elements appearing in the second text information and related elements in the first description information, and to use necessary attributes. It can be understood that the first semantic granularity, the second semantic granularity, and the third semantic granularity gradually increase.

[0101] For example, in one example, the second text information is "the person should be holding a baby", and the first description information of the second reference image is "a black-haired woman smiling under a gray umbrella with a white flower hanging on the umbrella". The second description information of the first semantic granularity generated by the text analysis large model is: "a woman holding a baby", the second description information of the second semantic granularity is: "a black-haired woman holding a baby under an umbrella", and the second description information of the third semantic granularity is: "a black-haired woman holding a baby, smiling under a gray umbrella". The second description information of the first semantic granularity only includes elements in the second text information, such as "person" and "baby"; the second description information of the second semantic granularity only includes the elements "person" and "baby", and the corresponding attributes "black hair" and "under an umbrella"; the second description information of the third semantic granularity includes the elements "person" and "baby", and the attribute "gray" of the element "umbrella" related to them in the first description information.

[0102] In an embodiment of the present disclosure, a text analysis large model is used to perform a first description generation task, and second description information of multiple semantic granularities is generated for the same input, thereby enriching the semantic hierarchy of the second description information, facilitating subsequent use of the second description information of multiple semantic granularities to determine the target image in a wider range, and helping to improve the accuracy of image search.

[0103] Figure 5 A schematic diagram of a scene for determining at least one target image according to an embodiment of the present disclosure is schematically shown.

[0104] like Figure 5 As shown, in embodiment 500, second text information 502 and second reference image 503 are determined according to whether input information 501 includes first text information and / or first reference image information.

[0105] After obtaining the first description information 504 for describing the second reference image 503, the first description information 504 and the second text information 502 are analyzed using the text analysis model M1 to obtain at least one second description information, such as second description information 1 505-1 ... second description information M 505-M. Using the second description information 1 505-1 ... second description information M505-M, at least one target image, such as target image 1 507-1 ..., can be screened from the search image library 506.

[0106] According to an embodiment of the present disclosure, for operation S220, a first description generation task is executed using a text analysis big model to perform semantic analysis on the second text information and the first description information to generate second description information of multiple semantic granularities, including: obtaining prompt information, wherein the prompt information includes multiple semantic granularities to be generated, and explanation information for each semantic granularity; and executing the first description generation task using a text analysis big model to perform semantic analysis on the second text information and the first description information based on each semantic granularity to generate second description information at each semantic granularity.

[0107] For example, prompt information A may be:

[0108] # Task description: You will be provided with a description about image search. The first description generation task is to combine the second text information with the information in the second reference image or the first description information to accurately search for images. # First description generation task: Based on the text instructions and reference image analysis, describe what the target image should look like. Provide three sentences to describe the target image, each sentence focusing on a different semantic granularity: (1) The first semantic granularity is the core element: it can be only the elements that appear in the second text information, without using attributes. (2) The second semantic granularity is enhanced details: only the elements that appear in the second text information are included, and necessary attributes are used from the second description information. (3) The third semantic granularity is comprehensive synthesis: including elements that appear in the second text information and related elements in the first description information, and using necessary attributes. I will give you a second text information and a first description information, and you need to complete the task based on this information. ### Query: Second text information [[instructions]], first description information [[reference image description]].

[0109] In this embodiment, [[instructions]] and [[reference image description]] are placeholders. "Core elements", "enhanced details" and "comprehensive synthesis" are the names of semantic granularities, and "may only include elements appearing in the second text information without using attributes" is the explanation information of the corresponding semantic granularity.

[0110] For another example, the prompt information may include not only the above content but also task examples so that the text analysis model can better perform the first description generation task.

[0111] In an embodiment of the present disclosure, by writing semantic granularity and explanation information of the semantic granularity in the prompt information, the first description generation task is performed using a text analysis big model to perform semantic analysis on the second text information and the first description information based on each semantic granularity, and generate second description information at each semantic granularity, thereby generating rich second description information.

[0112] by Figure 5For example, the text analysis model M1 can execute the first description generation task, and the obtained second description information 1 507 - 1 . . . belongs to multiple semantic granularities.

[0113] According to an embodiment of the present disclosure, for operation S220, the text analysis big model is used to sequentially execute the operation generation task and the second description generation task; using the text analysis big model, text analysis is performed on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: using the text analysis big model to execute the operation generation task to generate operation prompt information based on the difference between the first description information and the second text information, the operation prompt information including at least one operation of at least one operation type; and using the text analysis big model to execute the second description generation task to generate at least one second description information based on the operation prompt information and the first description information.

[0114] The text input by the user may contain multiple semantics. For example, the second text information includes multiple elements, multiple attributes, and multiple operations. When the second description information is generated directly based on the entire second text information using the text analysis model, the generated second description information may be inaccurate. Therefore, in this embodiment, the process of generating the second description information is divided into two tasks, the operation generation task and the second description generation task.

[0115] The operation generation task may be a classification task, which is used to compare the first description information and the second text information, and determine the operation to be performed on the first description information and the type of operation according to the difference between the two.

[0116] The operation prompt information may be a benchmark for executing the second description generation task, and is used to prompt the text analysis large model to execute the second description generation task and generate at least one second description information.

[0117] The operation types in the operation prompt information may include: addition type, removal type, modification type, comparison type, and retention type.

[0118] In one embodiment, when the text analysis large model sequentially executes the operation generation task and the second description generation task, the prompt information used may include the operation type and explanation information of the operation type.

[0119] For the second reference image described by the second description information, the explanatory information of the operation type can be: addition can be understood as introducing new elements or attributes into the second reference image; deletion can be understood as deleting certain elements or attributes from the second reference image; modification can be understood as changing the attributes of existing elements in the second reference image; comparison uses words such as "different", "same", "more" or "less" to compare the elements in the second reference image and the first text information; retention indicates that certain existing elements or attributes in the second reference image remain unchanged.

[0120] The operation prompt information obtained after executing the operation to generate the character may include second text information, in which the operation type is newly added.

[0121] For example, in one embodiment, the second text information is "the character should hold a baby", and the first description information of the second reference image is "a woman with black hair is smiling under a gray umbrella with a white flower hanging on it". The operation prompt information is "Add: woman holding a baby", the operation is to add "woman holding a baby" to the second reference image, and the operation type is "add". The "woman holding a baby" included in the operation prompt information contains the second text information "the character should hold a baby", which is a more specific and accurate description.

[0122] The second description generation task is similar to the first description generation task, but performs a specific operation according to the operation type specified in the operation prompt information, which will not be described in detail here.

[0123] In the embodiment of the present disclosure, the text analysis model is used to sequentially execute the operation generation task and the second description generation task to generate operation prompt information with clearer semantic meaning, and at least one second description information is generated based on the operation prompt information and the second text information. Since more clear and smaller-granular operations and operation types are extracted from the complex second text information through two tasks, the generated second description information better meets the image search requirements and helps to improve the image search accuracy.

[0124] Fig. 6A The following schematically shows a scenario diagram of generating second description information according to an embodiment of the present disclosure. Fig. 6A As shown, in embodiment 600A, the text analysis model M1 sequentially executes the operation generation task T1 and the second description generation task T2.

[0125] For example, the text analysis model M1 is used to perform the operation generation task T1 to generate the operation prompt information 603 according to the second text information 601 and the first description information 602. The operation prompt information 603 may include at least one operation of at least one operation type, such as operation 1 6031-1 ... operation n 6031-n of operation type 1 6031.

[0126] Taking operation 1 6031 - 1 as an example, the second description generation task T2 is performed using the text analysis model M1 to obtain a second description information 1 6041 according to operation 1 6031 - 1 and the first description information 602 .

[0127] In other embodiments, the operation prompt information may include only one operation of one operation type.

[0128] According to an embodiment of the present disclosure, when the second reference image is a blank reference image, the operation type of at least one operation included in the operation prompt information is an adding type.

[0129] For a single language modality image search task, the input information only includes the first text information, and the second reference image of the search information determined according to the input information is a blank reference image. At this time, the difference between the second text information and the first description information of the second reference image is the entire content of the second text information. Therefore, the elements, attributes, and operation types on the elements and / or attributes in the second text information are all added types.

[0130] For example, the second text information may be "a person holding a baby, the baby is wearing pink clothes", and the second reference image is a blank reference image. The operation prompt information obtained by executing the operation generation task may be: 1. Add: a person holding a baby; 2. Add: the baby is wearing pink clothes. The second description information generated according to the operation prompt information and the first description information may be "a person holding a baby wearing pink clothes".

[0131] In the case where the second text information is blank text information, the operation type of the operation included in the operation prompt information may be a retention type and / or a comparison type. For example, the second text information is blank text information, and the first description information of the second reference image may be "a person holding a baby, and the baby is wearing pink clothes". The operation prompt information obtained by executing the operation generation task may be: 1. Retention: a person holding a baby; 2. Retention: a baby is wearing pink clothes. The second description information generated according to the operation prompt information and the first description information may also be "a person holding a baby wearing pink clothes".

[0132] In the embodiments of the present disclosure, for a blank second reference image or second text information, the image search task can be performed normally using a large text analysis model, and a target image with semantics similar to a non-blank second text information or second reference image can be obtained, which has a wide range of usage scenarios.

[0133] According to an embodiment of the present disclosure, a text analysis big model is used to sequentially execute an operation generation task and a third description generation task, and the text analysis big model is used to perform text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: executing the operation generation task using the text analysis big model to generate operation prompt information based on the difference between the first description information and the second text information; and executing the third description generation task using the text analysis big model to generate second description information of multiple semantic granularities based on the operation prompt information and the first description information.

[0134] The operation generation task is similar to the above operation generation task, and the third description generation task is similar to the above second description generation task, generating the second description information according to the operation prompt information and the first description information; at the same time, when generating the second description information, the third description generation task generates the second description information of multiple semantic granularities similar to the first description generation task. The specific operation is as described above, which will not be repeated here.

[0135] When the text analysis large model sequentially executes the operation generation task and the third description generation task, the prompt information used may include multiple semantic granularities to be generated and explanation information for each semantic granularity; at the same time, it also includes the operation type and the explanation information of the operation type.

[0136] For example, the prompt information B used by the text analysis large model to sequentially execute the operation generation task and the third description generation task can be:

[0137] # Task description: You will be given a description about image search. The task is to combine the second text information with the information in the second reference image or the first description information to accurately search for images. You need to follow two steps to infer "what the target image looks like". # Step 1: Classify the given second text information into the following operation types and determine how it affects the second reference image. For each operation type, determine the elements or attributes of the second reference image that are affected. Operation types include: (1) Addition: Introducing new elements or attributes into the second reference image. Determine which existing element the addition is related to or where it should be placed. (2) Deletion: Delete certain elements from the second reference image. Determine which existing element was deleted. (3) Modification: Change the attributes of existing elements in the second reference image. Determine which elements are being modified and how they are modified. (4) Comparison: Use terms such as "different", "same", "more", or "less" to compare elements in the second reference image. Determine the elements and attributes being compared. (5) Preservation: Specify that certain existing elements in the second reference image remain unchanged. Make sure these elements are marked as included in the target image. # Step 2: ... . Among them, "#Step 2: ..." is similar to the #First description generation task in the above prompt information A, and will not be repeated here.

[0138] In an embodiment of the present disclosure, by sequentially executing the operation generation task and the third description generation task, the text analysis large model can generate more accurate and semantically richer second description information by performing tasks in a hierarchical manner, thereby improving the accuracy of the target image subsequently determined using the second description information with multiple semantic granularities.

[0139] Figure 6B A schematic diagram of a scenario for generating second description information according to another embodiment of the present disclosure is shown schematically. Figure 6B As shown, in embodiment 600B, the text analysis model M1 sequentially executes the operation generation task T1 and the third description generation task T3, wherein the manner of generating operation 1 6031-1…operation n 6031-n is similar to that of embodiment 600A and will not be repeated here.

[0140] Taking operation 1 6031-1 as an example, by using the text analysis model M1 to perform the third description generation task T3, multiple second description information of multiple semantic dimensions can be obtained, such as second description information 1 6042-1...second description information m 6042-m.

[0141] According to an embodiment of the present disclosure, for an operation generation task, generating operation prompt information according to the difference between the first description information and the second text information includes: extracting at least one operation from the second text information and determining the operation type according to the difference between the first description information and the second text information; or, extracting at least one initial operation from the second text information and determining the operation type according to the difference between the first description information and the second text information; and rewriting the initial operation using the first description information to obtain the operation.

[0142] The operation generation task may include an operation extraction task and a classification task. The operation extraction task is used to compare the first description information and the second text information, and extract one or more operations to be executed based on the difference between the two; then classify the extracted operations, generate the operation type of the corresponding operation, and obtain operation prompt information.

[0143] The operation generation task may also include an operation extraction task, a classification task and a rewriting task. By executing the above operation extraction task and classification task, the corresponding initial operation and operation type are generated; then, the initial operation is rewritten according to the first description information to obtain an operation.

[0144] For example, if the second text information is "the person should be holding a baby" and the first description information of the second reference image is "a woman with black hair is smiling under a gray umbrella with a white flower hanging on it", the operation prompt information before rewriting is: 1. Add: the person is holding a baby. The initial operation "the person is holding a baby" is rewritten using the person "woman" in the first description information, and the operation prompt information is: 1. Add: the woman is holding a baby.

[0145] In the embodiments of the present disclosure, the text analysis model can generate operation prompt information with clearer instructions by rewriting the initial operation, so as to subsequently generate more accurate second description information based on the operation prompt information and the first description information.

[0146] According to an embodiment of the present disclosure, for operation S230, at least one target image is determined based on at least one second description information, including: determining the similarity between the second description information and third description information used to describe the candidate image, wherein the search image library includes multiple candidate images; for each candidate image, determining a first comprehensive similarity based on the similarity between the second description information and each second description information; and determining at least one target image from the multiple candidate images based on the first comprehensive similarity.

[0147] The user-specified or default search image library includes a plurality of candidate images. At least one target image can be screened out from the plurality of candidate images in the search image library according to the second description information.

[0148] For example, the search image library can store multiple candidate images and third description information for describing each candidate image. For the second description information and the third description information of the language modality, the similarity can be calculated using an existing similarity algorithm. Then, for each candidate image, the similarity between the third description information and each second description information is weighted and summed to obtain a first comprehensive similarity for each candidate image. Alternatively, the first comprehensive similarity for each candidate image is obtained based on the number of similarities that reach a certain threshold. Thereafter, based on the first comprehensive similarity, at least one target image is directly determined from the multiple candidate images, such as selecting a candidate image with a high first comprehensive similarity as the target image.

[0149] In the embodiments of the present disclosure, since the third description information of the language modality is used and the similarity between the third description information and the second description information is calculated, the first comprehensive similarity obtained by combining the above similarities can determine a target image with more accurate semantics in the same modality, and the search accuracy is higher.

[0150] According to an embodiment of the present disclosure, for operation S230, at least one target image is determined based on at least one second description information, including: obtaining image coding features of each candidate image in the search image library; for each candidate image, determining a second comprehensive similarity based on the similarity between the image coding features and the text coding features of each second description information; and determining at least one target image from multiple candidate images based on the second comprehensive similarity.

[0151] For example, the second description information of the language modality can be encoded using a text encoding model to obtain text encoding features, such as a text vector; and the candidate image of the visual modality can be encoded using an image encoding model to obtain image encoding features, such as an image vector.

[0152] Alternatively, a multimodal large model may be used to encode the second description information and the candidate image respectively, to obtain text encoding features and image encoding features respectively. For example, a vision-language model (VLM) composed of a visual encoder and a language encoder is used to encode the second description information and the candidate image respectively using the internal language encoder and image encoder.

[0153] The existing vector similarity algorithm can be used to calculate the similarity between each image coding feature and the text coding feature of each second description information. For a single candidate image, the similarity between the image coding feature and each second description information is weighted and summed to obtain the second comprehensive similarity of each candidate image. Alternatively, the second comprehensive similarity of each candidate image is obtained based on the number of similarities that reach a certain threshold. Afterwards, at least one target image is directly determined from multiple candidate images based on the second comprehensive similarity, such as selecting a candidate image with a high second comprehensive similarity as the target image.

[0154] In an embodiment of the present disclosure, when determining a target image, the similarity between the image coding features of the candidate image and the text coding features of the second description information is used. Therefore, the selected target image can meet the image search requirements from a visual perspective, thereby ensuring the accuracy of the target image from a visual perspective.

[0155] According to an embodiment of the present disclosure, the method also includes: for each candidate image, determining a third comprehensive similarity based on the similarity between the third description information and each second description information, and the similarity between the image coding feature and the text coding feature of each second description information; and determining at least one target image from multiple candidate images based on the third comprehensive similarity.

[0156] Referring to the above calculation method, the similarity between the third description information and the second description information, and the similarity between the image coding feature and the text coding feature can be obtained.

[0157] For ease of description, the above two similarities are referred to as text-text similarity and text-image similarity. For each candidate image, the text-text similarity and text-image similarity between the candidate image and each second description information can be integrated to obtain a third comprehensive similarity. Then, at least one target image is directly selected from the multiple candidate images according to the third comprehensive similarity.

[0158] In one embodiment, the text-to-text similarity and text-to-image similarity corresponding to each second description information may be weighted and summed according to the semantic granularity; and the weighted summation results of multiple semantic granularities are added and divided by the number of semantic granularities to obtain a third comprehensive similarity. For example, when the text-to-text similarity and text-to-image similarity corresponding to each second description information are weighted and summed, a parameter To control the weight between the two so that the sum of their weights is 1.

[0159] In the embodiments of the present disclosure, a third comprehensive similarity is obtained by integrating text-text similarity and text-image similarity, so that the target image determined according to the third comprehensive similarity can better meet the image search requirements from both text and image perspectives, enhance robustness, and improve the accuracy of image search.

[0160] Figure 7 The following schematically shows a scene diagram of determining a target image from a search image library according to a specific embodiment of the present disclosure. Figure 7 As shown, in the embodiment 700, the second text information 702 and the second reference image 703 are determined according to whether the input information 701 includes the first text information and / or the first reference image information. After obtaining the first description information 704 for describing the second reference image 703, the first description information 704 and the second text information 702 are subjected to text analysis using the text analysis model M1 to obtain at least one second description information, such as second description information 1 705-1 ... second description information M 705-M.

[0161] The search image library 706 includes a plurality of candidate images, such as candidate image 1 706-1. Taking candidate image 1 706-1 as an example, for the third description information 708-1 used to describe candidate image 1 706-1, the similarity between the third description information 708-1 and each second description information can be calculated, and based on the similarity between the third description information 708-1 and each second description information, a comprehensive similarity 709-1 corresponding to candidate image 1 706-1 is obtained. For each candidate image, the corresponding comprehensive similarity can be obtained by using the above method.

[0162] Afterwards, at least one target image, such as target image 1 707-1 . . . , is determined from the search image library 706 according to the comprehensive similarity of each candidate image.

[0163] The comprehensive similarity may be a first comprehensive similarity, a second comprehensive similarity, or a third comprehensive similarity.

[0164] According to an embodiment of the present disclosure, the method further includes: converting the second reference image into first description information using the description generation large model; and converting the candidate image into third description information using the description generation large model.

[0165] The description generation large model may be a large multimodal model (Large Multimodal Models, LMMs), which is used to convert the input of the visual modality into the output of the language modality. For example, the input second image is converted into the first description information, and the candidate image is converted into the third description information. The description generation large model may be a pre-trained large multimodal model.

[0166] In the embodiments of the present disclosure, by describing and generating a large model to generate description information of a corresponding image, the operation can be simplified and the processing flow of the entire image search task can be more intelligent.

[0167] To facilitate understanding of the present disclosure, a specific embodiment will be taken as an example to illustrate the image search process. Figure 8 A schematic diagram of a scene for determining a target image according to a specific embodiment of the present disclosure is schematically shown.

[0168] like Figure 8 As shown, the input information may include the first text information "-reference image description / previous round of dialogue information: a black dog and a brown dog are sitting next to a gray refrigerator in the kitchen. -text instructions / text feedback: only black dogs are needed; a woman in a black shirt is cooking on the stove"; the input information also includes a first reference image, such as image A. In this embodiment, the input image has a first reference image, so the second reference image in the search information, that is, image A, can also be obtained by express.

[0169] The second reference image can be generated by using the description Captioner Convert to the first description information , the above generation process can be expressed by formula (1):

[0170] (1)

[0171] like Figure 8As shown, the text analysis model Reasoner is used to sequentially perform the operation generation task and the third description generation task based on the prompt information Prompt1 to generate three second description information of semantic granularity 1 to semantic granularity 3. For example, in the second step, "Semantic granularity 1: A woman in a black shirt is cooking on the stove, and there is a black dog sitting next to her. Semantic granularity 2: A woman in a black shirt is cooking on the stove in the kitchen, and there is a black dog sitting next to her. Semantic granularity 3: A woman in a black shirt is cooking on the stove in the kitchen, and there is a black dog sitting next to the gray refrigerator." Figure 8 In the example, the operation prompt information generated by the operation generation task can be: "1. Add: Add a woman in a black shirt. 2. Add: The woman is cooking. 3. Delete: Delete the brown dog. 4. Keep: Keep the black dog." of the first step.

[0172] The process of generating at least one second description information using the text analysis model Reasoner can be expressed by the following formula (2):

[0173] (2)

[0174] in, is the second text message, Representing M operations A collection of They are the second description information of three semantic granularities CE, ED, and CS respectively.

[0175] The second description information is generated by using the visual language model VLM After vectorization, the obtained text encoding features can be vectorized indicates that, and , d is the vector length, where .

[0176] In addition, in another branch, for the M candidate images in the search image library, such as image 1, image 2, ... image M, the third description information of each candidate image can be generated in advance by the description generation model Captioner. , describing the third description information of the corresponding image output by the large model Captioner, After that, use The third description information of each candidate image is vectorized to obtain text encoding features (text vectors), and each candidate image is vectorized to obtain image encoding features (image vectors). The above conversion process can be seen in formulas (3) and (4):

[0177] (3)

[0178] (4)

[0179] in, Indicates that the length of each third description information is The text vector of Indicates that the length of each candidate image is The image vector of represents the matrix form of the text vector of all third-party description information, and Represents all image vectors in matrix form.

[0180] After the text analysis model Reasoner generates at least one second description information, the similarity between the text encoding feature of the second description information and the image encoding feature of the candidate image and the text encoding feature of the third description information of the candidate image is calculated to obtain the third comprehensive similarity of each candidate image. The above process can be expressed by formula (5):

[0181] (5)

[0182] in, is the vector form of the third comprehensive similarity of multiple candidate images, , represents the cosine similarity function, It is the abbreviation for text encoding features at three semantic granularities. For adjustment and The weight between .

[0183] The process of determining at least one target image according to the third comprehensive similarity is as shown in formula (6):

[0184] (6)

[0185] in, For N candidate images Image search sequence in descending order.

[0186] In one embodiment, you can select Output as the final target image.

[0187] Fig. 9 A block diagram of an image search device according to a specific embodiment of the present disclosure is schematically shown.

[0188] like Fig. 9 As shown, the image search device 900 includes: a first determination module 910 , a generation module 920 , and a second determination module 930 .

[0189] The first determination module 910 is used to determine multimodal search information according to input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image.

[0190] The generating module 920 is used to perform text analysis on the second text information and the first description information used to describe the second reference image by using the text analysis large model, so as to generate at least one second description information.

[0191] The second determination module 930 is configured to determine at least one target image according to at least one second description information, wherein each target image is determined according to at least one second description information.

[0192] According to an embodiment of the present disclosure, the first determination module 910 includes: a first determination submodule, which is used to determine the first text information as the second text information when the input information includes the first text information, and obtain the second reference image, wherein the second reference image includes a blank reference image. A second determination submodule, which is used to determine the first reference image as the second reference image when the input information includes the first reference image, and obtain the second text information, wherein the second text information includes blank text information. A third determination submodule, which is used to determine the first text information and the first reference image as the second text information and the second reference image, respectively, when the input information includes the first text information and the first reference image.

[0193] According to an embodiment of the present disclosure, the image search device 900 further includes: a third determination module, which is used to determine the indicated image library as the search image library when the input information includes the indicated image library, so as to determine the target image from the search image library. A fourth determination module is used to determine the reference image library as the search image library when the input information does not include the indicated image library.

[0194] According to an embodiment of the present disclosure, input information is obtained based on at least one of the following methods: determining a first text information and / or a first reference image based on input text and input images in at least one round of conversation; determining a first reference image and / or a first text information based on at least one output image and / or input text in at least one round of conversation, wherein the output image is a target image determined and output based on input information before the output image.

[0195] According to an embodiment of the present disclosure, a text analysis big model is used to perform a first description generation task, and a generation module 920 includes: a first generation submodule, used to perform the first description generation task using a text analysis big model to perform a semantic analysis on the second text information and the first description information, and generate second description information of multiple semantic granularities; wherein the semantic granularity is related to the number and / or attributes of elements in the second description information, and the elements are extracted from the second text information and / or the first description information.

[0196] According to an embodiment of the present disclosure, the first generation submodule includes: an acquisition unit, used to acquire prompt information, wherein the prompt information includes multiple semantic granularities to be generated, and explanation information for each semantic granularity; a first generation unit, used to perform a first description generation task using a text analysis big model, so as to perform semantic analysis on the second text information and the first description information based on each semantic granularity, and generate at least one second description information under each semantic granularity.

[0197] According to an embodiment of the present disclosure, the text analysis big model is used to sequentially execute the operation generation task and the second description generation task; the generation module 920 includes: a second generation submodule, used to use the text analysis big model to execute the operation generation task, so as to generate operation prompt information according to the difference between the first description information and the second text information, and the operation prompt information includes at least one operation of at least one operation type; and a third generation submodule, used to use the text analysis big model to execute the second description generation task, so as to generate at least one second description information according to the operation prompt information and the first description information.

[0198] According to an embodiment of the present disclosure, it also includes: when the second reference image is a blank reference image, the operation type of at least one operation included in the operation prompt information is an adding type.

[0199] According to an embodiment of the present disclosure, the text analysis big model is used to sequentially execute the operation generation task and the third description generation task, and the generation module 920 includes: a fourth generation submodule, used to use the text analysis big model to execute the operation generation task to generate operation prompt information according to the difference between the first description information and the second text information; and a fifth generation submodule, used to use the text analysis big model to execute the third description generation task to generate second description information of multiple semantic granularities according to the operation prompt information and the first description information.

[0200] According to an embodiment of the present disclosure, the fourth generation submodule includes: a second generation unit, used to extract at least one operation from the second text information and determine the operation type based on the difference between the first description information and the second text information; or, a third generation unit, used to extract at least one initial operation from the second text information and determine the operation type based on the difference between the first description information and the second text information; and rewrite the initial operation using the first description information to obtain the operation.

[0201] According to an embodiment of the present disclosure, the second determination module 930 includes: a fourth determination submodule, used to determine the similarity between the second description information and the third description information used to describe the candidate image, wherein the search image library includes multiple candidate images; a fifth determination submodule, used to determine, for each candidate image, a first comprehensive similarity based on the similarity between the second description information and each candidate image; and a sixth determination submodule, used to determine at least one target image from the multiple candidate images based on the first comprehensive similarity.

[0202] According to an embodiment of the present disclosure, the second determination module 930 includes: an acquisition submodule, used to acquire image coding features of each candidate image in the search image library; a seventh determination submodule, used to determine, for each candidate image, a second comprehensive similarity based on the similarity between the image coding features and the text coding features of each second description information; and an eighth determination submodule, used to determine at least one target image from multiple candidate images based on the second comprehensive similarity.

[0203] According to an embodiment of the present disclosure, the second determination module 930 also includes: a ninth determination submodule, for determining, for each candidate image, a third comprehensive similarity based on the similarity between the third description information and each second description information, and the similarity between the image coding feature and the text coding feature of each second description information; and a tenth determination submodule, for determining at least one target image from multiple candidate images based on the third comprehensive similarity.

[0204] According to an embodiment of the present disclosure, the image search device 900 also includes: a first conversion module, used to convert the second reference image into first description information using the description generation model; and a second conversion module, used to convert the candidate image into third description information using the description generation model.

[0205] Fig.10 The structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is schematically shown.

[0206] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig.10As shown, the AI ​​agent 1000 may include five core modules: an input module 1010 , a control module 1020 , a storage module 1030 , a calculation module 1040 and an output module 1050 .

[0207] The input module 1010 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 1000. The input module 1010 is the primary link for the AI ​​agent 1000 to interact with the outside world. It enables the AI ​​agent 1000 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0208] In an example, the input module 1010 may input the input information described above, the first text information, and the first reference image.

[0209] In the example, the control module 1020 is the core support for the AI ​​agent 1000 to handle complex tasks. The control module 1020 can execute the image search method described above.

[0210] In the example, the control module 1020 will continuously interact with the storage module 1030, the computing module 1040 and / or the output module 1050 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 1020 acts as a single initiator to initiate communication with the storage module 1030, the computing module 1040 and / or the output module 1050, and there is no communication coupling between the storage module 1030, the computing module 1040 and the output module 1050.

[0211] In the example, the performance of the control module 1020 may be closely related to the large model on which the AI ​​agent 1000 is based. In order to give full play to the capabilities of the large model, the internal structure of the control module 1020 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0212] The storage module 1030 may be responsible for memorizing information such as historical conversations, event flows, etc. The aforementioned prompt information, the third description information of the candidate images, etc. may be included in the storage module 1030 .

[0213] The operation module 1040 can be regarded as a predefined tool library. As described above, the controls for text encoding and image encoding can be included in the operation module 1040.

[0214] In an example, the output module 1050 may output at least one target image described above.

[0215] The AI ​​agent 1000 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0216] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0217] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0218] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0219] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0220] Fig.11 A block diagram of an electronic device 1100 suitable for implementing an image search method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0221] like Fig.11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 to a random access memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0222] A number of components in the device 1100 are connected to an input / output (I / O) interface 1105, including: an input unit 1106, such as a keyboard, a mouse, etc.; an output unit 1107, such as various types of displays, speakers, etc.; a storage unit 1108, such as a disk, an optical disk, etc.; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1109 allows the device 1100 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0223] The computing unit 1101 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1101 performs the various methods and processes described above, such as the image search method. For example, in some embodiments, the image search method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1100 via the ROM 1102 and / or the communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the image search method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the image search method in any other appropriate manner (e.g., by means of firmware).

[0224] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0225] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0226] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0227] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0228] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0229] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0230] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0231] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image search method, comprising: Determining multimodal search information according to input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image; Using a text analysis large model, performing text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information; and At least one target image is determined according to at least one of the second description information, wherein each of the target images is determined according to at least one of the second description information.

2. The method according to claim 1, wherein: Determining multimodal search information according to input information for image search includes: In a case where the input information includes the first text information, determining the first text information as the second text information, and acquiring the second reference image, wherein the second reference image includes a blank reference image; In a case where the input information includes the first reference image, determining the first reference image as the second reference image, and acquiring the second text information, wherein the second text information includes blank text information; In a case where the input information includes the first text information and the first reference image, the first text information and the first reference image are determined as the second text information and the second reference image, respectively.

3. The method according to claim 1 or 2, further comprising: In a case where the input information includes an indication image library, determining the indication image library as a search image library, so as to determine the target image from the search image library; In a case where the input information does not include the indicated image library, a reference image library is determined as the search image library.

4. The method according to claim 1, wherein the input information is obtained based on at least one of the following methods: Determine the first text information and / or the first reference image based on input text and input image in at least one round of conversation; The first reference image and / or the first text information is determined based on at least one output image and / or the input text in at least one round of the conversation, wherein: The output image is the target image determined and output according to the input information before the output image.

5. The method according to any one of claims 1 to 4, wherein: The text analysis big model is used to perform the first description generation task, and the text analysis big model is used to perform text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: Using a large text analysis model to perform the first description generation task, so as to perform semantic analysis on the second text information and the first description information, and generate the second description information of multiple semantic granularities; The semantic granularity is related to the number and / or attributes of elements in the second description information, and the elements are extracted from the second text information and / or the first description information.

6. The method according to claim 5, wherein: The using of the text analysis big model to perform the first description generation task to perform semantic analysis on the second text information and the first description information to generate the second description information of multiple semantic granularities includes: Acquiring prompt information, wherein the prompt information includes a plurality of the semantic granularities to be generated and explanation information for each of the semantic granularities; and The first description generation task is performed using the text analysis big model to perform semantic analysis on the second text information and the first description information based on each of the semantic granularities to generate second description information at each of the semantic granularities.

7. The method according to any one of claims 1 to 5, wherein: The text analysis big model is used to sequentially execute the operation generation task and the second description generation task; the text analysis big model is used to perform text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: Executing the operation generation task using a text analysis big model to generate operation prompt information according to a difference between the first description information and the second text information, the operation prompt information including at least one operation of at least one operation type; and The second description generation task is performed using the text analysis big model to generate at least one second description information according to the operation prompt information and the first description information.

8. The method according to claim 7, further comprising: In the case where the second reference image is the blank reference image, the operation type of at least one operation included in the operation prompt information is an adding type.

9. The method according to claim 1, wherein the text analysis big model is used to sequentially execute the operation generation task and the third description generation task, wherein the text analysis big model is used to perform text analysis on the second text information and the first description information used to describe the second reference image to generate at least one second description information, including: Utilizing the text analysis big model to perform the operation generation task, so as to generate operation prompt information according to the difference between the first description information and the second text information; as well as The third description generation task is performed using a text analysis large model to generate the second description information of multiple semantic granularities based on the operation prompt information and the first description information.

10. The method according to any one of claims 7 to 9, wherein: The generating operation prompt information according to the difference between the first description information and the second text information includes: extracting at least one operation from the second text information and determining the operation type according to the difference between the first description information and the second text information; or, According to the difference between the first description information and the second text information, at least one initial operation is extracted from the second text information and the operation type is determined; and the initial operation is rewritten using the first description information to obtain the operation.

11. The method according to any one of claims 1 to 10, wherein: The determining at least one target image according to at least one of the second description information comprises: determining a similarity between the second description information and third description information used to describe a candidate image, wherein the search image library includes a plurality of the candidate images; For each of the candidate images, determine a first comprehensive similarity according to the similarity between the candidate images and each of the second description information; and At least one target image is determined from the plurality of candidate images according to the first comprehensive similarity.

12. The method according to any one of claims 1 to 10, wherein: The determining at least one target image according to at least one of the second description information comprises: Obtaining image coding features of each candidate image in the search image library; For each of the candidate images, determining a second comprehensive similarity according to a similarity between the image encoding feature and each text encoding feature of the second description information; and At least one target image is determined from the plurality of candidate images according to the second comprehensive similarity.

13. The method according to claim 11 or 12, further comprising: For each of the candidate images, determining a third comprehensive similarity according to a similarity between the third description information and each of the second description information, and a similarity between the image encoding feature and each of the text encoding features of the second description information; and At least one target image is determined from the plurality of candidate images according to the third comprehensive similarity.

14. The method according to any one of claims 1 to 13, further comprising: Converting the second reference image into the first description information using a description generation model; as well as The candidate image is converted into the third description information using the description generation model.

15. An image search device, comprising: A first determining module, configured to determine multimodal search information according to input information for image search, wherein the input information includes first text information and / or a first reference image, and the search information includes second text information and a second reference image; a generating module, configured to perform text analysis on the second text information and the first description information for describing the second reference image by using a text analysis large model, and generate at least one second description information; and The second determination module is used to determine at least one target image according to at least one second description information, wherein each target image is determined according to at least one second description information.

16. The device according to claim 15, wherein: The first determining module comprises: A first determination submodule, configured to determine the first text information as the second text information and obtain the second reference image when the input information includes the first text information, wherein the second reference image includes a blank reference image; a second determining submodule, configured to, when the input information includes the first reference image, determine the first reference image as the second reference image, and obtain the second text information, wherein the second text information includes blank text information; The third determining submodule is configured to, when the input information includes the first text information and the first reference image, determine the first text information and the first reference image as the second text information and the second reference image, respectively.

17. The device according to claim 15 or 16, further comprising: a third determining module, configured to, when the input information includes an indication image library, determine the indication image library as a search image library, so as to determine the target image from the search image library; The fourth determining module is configured to determine a reference image library as the search image library when the input information does not include the indication image library.

18. The device according to claim 15, wherein the input information is obtained based on at least one of the following methods: Determine the first text information and / or the first reference image based on input text and input image in at least one round of conversation; The first reference image and / or the first text information is determined based on at least one output image and / or the input text in at least one round of the conversation, wherein: The output image is the target image determined and output according to the input information before the output image.

19. The device according to any one of claims 15 to 18, wherein: The text analysis model is used to perform the first description generation task, and the generation module includes: A first generating submodule, configured to use a text analysis macromodel to perform the first description generating task, so as to perform semantic analysis on the second text information and the first description information, and generate the second description information of a plurality of semantic granularities; The semantic granularity is related to the number and / or attributes of elements in the second description information, and the elements are extracted from the second text information and / or the first description information.

20. The device according to claim 19, wherein The first generation submodule comprises: an acquiring unit, configured to acquire prompt information, wherein the prompt information includes a plurality of the semantic granularities to be generated and explanation information for each of the semantic granularities; and The first generating unit is used to use the text analysis big model to execute the first description generating task, so as to perform semantic analysis on the second text information and the first description information based on each semantic granularity, and generate at least one second description information under each semantic granularity.

21. The device according to any one of claims 15 to 19, wherein: The text analysis large model is used to sequentially execute the operation generation task and the second description generation task; The generation module comprises: a second generation submodule, configured to perform the operation generation task by using the text analysis macromodel to generate operation prompt information according to a difference between the first description information and the second text information, wherein the operation prompt information includes at least one operation of at least one operation type; and The third generation submodule is used to use the text analysis model to perform the second description generation task to generate at least one second description information according to the operation prompt information and the first description information.

22. The apparatus according to claim 21, further comprising: In the case where the second reference image is the blank reference image, the operation type of at least one operation included in the operation prompt information is an adding type.

23. The apparatus according to claim 15, wherein the text analysis model is used to sequentially execute the operation generation task and the third description generation task, and the generation module comprises: a fourth generation submodule, configured to perform the operation generation task using the text analysis macromodel, so as to generate operation prompt information according to the difference between the first description information and the second text information; as well as The fifth generation submodule is used to use the text analysis model to perform the third description generation task to generate the second description information of multiple semantic granularities based on the operation prompt information and the first description information.

24. The device according to any one of claims 21 to 23, wherein: The fourth generation submodule comprises: a second generating unit, configured to extract at least one operation from the second text information and determine the operation type according to a difference between the first description information and the second text information; or The third generating unit is used to extract at least one initial operation from the second text information and determine the operation type according to the difference between the first description information and the second text information; and rewrite the initial operation using the first description information to obtain the operation.

25. The device according to any one of claims 15 to 24, wherein: The second determining module comprises: a fourth determination submodule, configured to determine a similarity between the second description information and third description information used to describe a candidate image, wherein the search image library includes a plurality of the candidate images; a fifth determination submodule, configured to determine, for each of the candidate images, a first comprehensive similarity according to a similarity between the candidate images and each of the second description information; and The sixth determination submodule is used to determine at least one of the target images from the plurality of candidate images according to the first comprehensive similarity.

26. The device according to any one of claims 15 to 24, wherein: The second determining module comprises: An acquisition submodule, used to acquire image coding features of each candidate image in the search image library; a seventh determination submodule, configured to determine, for each candidate image, a second comprehensive similarity according to a similarity between the image encoding feature and each text encoding feature of the second description information; and An eighth determination submodule is used to determine at least one of the target images from the plurality of candidate images according to the second comprehensive similarity.

27. The apparatus according to claim 25 or 26, wherein the second determining module further comprises: a ninth determination submodule, configured to determine, for each of the candidate images, a third comprehensive similarity according to a similarity between the third description information and each of the second description information, and a similarity between the image encoding feature and the text encoding feature of each of the second description information; as well as The tenth determination submodule is used to determine at least one of the target images from the plurality of candidate images according to the third comprehensive similarity.

28. The device according to any one of claims 15 to 27, further comprising: A first conversion module, configured to convert the second reference image into the first description information by using a description generation model; as well as The second conversion module is used to convert the candidate image into the third description information by using the description to generate a large model.

29. An agent configured to execute the method according to any one of claims 1 to 14.

30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 14.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

32. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images