Method for picture retrieval and related apparatus

Abstract

A method for picture retrieval, comprising: on the basis of a first picture and first text from a client, a cloud generates a plurality of preview pictures, the first text being used for adjusting the first picture, and then sends the plurality of preview pictures to the client; on the basis of operation information from the client, the cloud retrieves a second picture associated with a target picture, the target picture being included in the plurality of preview pictures; and the cloud sends the second picture to the client.

Description

A method and related device for image retrieval

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on September 27, 2023, with application number 202311287307.X and application name “A method and related device for image retrieval”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of cloud computing technology, and in particular to a method and related device for image retrieval. Background Art

[0003] Retrieval is also called search. On a search engine or shopping platform, after the user enters keywords in the search box, the search platform will return the corresponding search results.

[0004] Currently, text search is the most commonly used method, but some search platforms also offer image search. Image search is a method of searching based on a user-entered image and returning images similar to the input image, also known as image search.

[0005] Although image retrieval can narrow the search scope compared to text retrieval, it will still retrieve many images. Users need to browse the returned images one by one to find the desired image, and accurate retrieval cannot be achieved.

[0006] Summary of the Invention

[0007] This application provides a method for image retrieval to improve the accuracy of image retrieval. This application also provides corresponding devices, systems, computer-readable storage media, and computer program products.

[0008] In a first aspect, the present application provides a method for image retrieval, comprising: generating multiple preview images based on a first image and a first text from a client, wherein the first text is used to adjust the first image; sending the multiple preview images to the client; retrieving a second image associated with a target image based on operation information from the client, wherein the target image is included in the multiple preview images; and sending the second image to the client.

[0009] In this application, the cloud can be software or services on a cloud platform, or software or services deployed on nodes in a network, such as edge nodes. The cloud can run on independent physical machines or on virtualized resources.

[0010] In this application, the client can be a terminal device or an application, for example, an application running on a terminal device for a user to use. The application can be a search engine, a response application, or a shopping application.

[0011] In this application, the user can input the first picture and the first text on the client, and use the first text to guide the cloud to adjust the first picture, thereby obtaining multiple preview pictures. The user can select a suitable target picture from the preview pictures, and the cloud will search based on the target picture, which can narrow the search scope and improve the accuracy of image retrieval.

[0012] In one possible implementation, the above steps of generating multiple preview images based on the first image and the first text from the client include: expanding the first text to obtain multiple extended texts; and generating a preview image corresponding to each extended text based on each extended text and the first image.

[0013] In this possible implementation, the cloud can expand on the content of the first text. For example, if the first text is about placing a flower pattern on a first image, the text can be expanded around the type of flower, the location of the flower on the first image, etc. By expanding the text, the resulting preview image can be provided from multiple angles to meet the user's requirements in the first text, thereby increasing the richness of the preview image within the user's requirements.

[0014] In a possible implementation, the multiple extended texts are texts in a text design database whose similarity to the first text satisfies a first condition.

[0015] In this possible implementation, the text design database can be pre-built and continuously updated, and the texts in the text design database can be collected based on historical text retrieval records. In this application, the text design database can be searched for texts with high similarity to the first text through the first text. The first condition can be a threshold value. Taking the similarity value in the range of 0 to 1 as an example, the threshold value can be 0.5 or other numerical values. Taking 0.5 as an example, texts with a similarity greater than 0.5 can be understood as texts whose similarity to the first text meets the first condition. The first condition can also be a restriction on the number of texts, such as: the top N texts with the highest similarity to the first text are texts whose similarity to the first text meets the first condition. It can be seen from this possible implementation that better extended texts can be obtained through the text design database, thereby improving user satisfaction with preview images.

[0016] In a possible implementation, the multiple extended texts are texts in a text design database whose similarity and design popularity with the first text meet a second condition.

[0017] In this possible implementation, a text design database can be searched for texts with high similarity to the first text using the first text. The popularity of these highly similar texts can then be determined. The similarity and popularity can then be combined to determine the extension text. Each text in the text design database can have its popularity calculated in advance based on, for example, the number of times it has been used. The popularity can be a value between 0 and 1. The second condition can include one or two thresholds. If there is only one threshold, the text similarity and popularity can be weighted to obtain a weighted value (the weight of similarity is generally greater than the popularity). If this weighted value is greater than the threshold, the text meets the second condition and is an extension text of the first text. If the second condition includes two thresholds, one threshold can be a similarity threshold, such as 0.5; the other threshold can be a threshold for the number of extension texts, such as N, where N is a positive integer. If the similarity of the found text to the first text is greater than 0.5, the extension texts with a similarity greater than 0.5 can be sorted by popularity, and the N extension texts with the highest popularity are selected as the final extension texts. In this application, the accuracy of the expanded text can be improved through similarity and popularity.

[0018] In a possible implementation, the second image is an image retrieved from an image vector database.

[0019] In this possible implementation, the image vector database can be pre-built and continuously updated, and the images in the image vector database can be collected from the Internet and continuously updated. Of course, the second image can also be searched from the Internet. In this application, by retrieving the second image from the image vector database, the speed of second image retrieval can be improved.

[0020] In one possible implementation, the method is applied to a response system based on a large language model (LLM); the operation information includes a target image, or the operation information includes a target image and a second text.

[0021] In this possible implementation, if the preview image contains a target image that the user is satisfied with, image retrieval can be performed based on the target image. Of course, retrieval can also be performed based on the target image and the extended text corresponding to the target image. If the preview image does not contain an image that the user is satisfied with, the target image can be used as the input image for the second image editing, and the second text can guide the cloud to edit the target image, and further provide the user with a second wave of preview images until the user selects a satisfactory image. From this possible implementation, it can be seen that in the response system, different models can be called through LLM to implement different functions such as image retrieval or image editing, thereby enriching the functions of the response system.

[0022] In one possible implementation, if the operation information includes a target image, correspondingly, the above step of retrieving a second image associated with the target image includes calling an image editing retrieval model through LLM, and retrieving the associated second image based on the extended text corresponding to the target image and the target image.

[0023] In this possible implementation, the image editing retrieval model retrieves the second image based on the extended text corresponding to the target image and the target image, which can improve the accuracy of the retrieved second image.

[0024] In one possible implementation, if the second text indicates that the target image meets the user's expectations, correspondingly, the above step: retrieving the second image associated with the target image includes: calling the image editing retrieval model through LLM, and retrieving the associated second image based on the extended text corresponding to the target image, the second text and the target image.

[0025] In this possible implementation, the image editing retrieval model retrieves the second image based on the extended text corresponding to the target image, the second text, and the target image, which can improve the accuracy of the retrieved second image.

[0026] In one possible implementation, if the second text indicates that the target image does not meet the user's expectations, correspondingly, the above-mentioned step of retrieving the second image associated with the target image includes: calling the image editing generation model through LLM to generate multiple preview images associated with the target image and the second text. The image editing generation model can obtain multiple preview images associated with the target image and the second text based on the multiple extended texts obtained by expanding the second text and the target image to determine the second image.

[0027] In this possible implementation, the image editing generation model refers to a model that can use the second text and the target image to further generate the next wave of preview images. The generation process can be understood by referring to the process of generating preview images from the first text and the first image. This multiple rounds of preview image generation can ensure that users obtain satisfactory preview images, thereby improving the accuracy of the retrieved second image.

[0028] In a possible implementation, the LLM-based answering system further includes a text analysis model, which is used to analyze the second text to determine whether the target image meets the user's expectations.

[0029] In this possible implementation, the second text can instruct the LLM to call the image editing retrieval model or the image editing generation model. The text analysis model can analyze the second text to determine whether the user is satisfied with the target image. If not, the LLM is instructed to call the image editing generation model to generate the next wave of preview images. If satisfied, the LLM is instructed to call the image editing retrieval model to generate the second image. This improves the efficiency of communication with the user.

[0030] In a possible implementation, when receiving the second picture from the cloud, the method further includes: receiving link information associated with the second picture from the cloud.

[0031] In this possible implementation, the link information associated with the second picture may be description information of the product in the second picture and a purchase link, etc. In this way, the efficiency of communication with the user can be improved.

[0032] A second aspect of the present application provides a method for image retrieval, including: sending a first image and a first text input by a user to a cloud, where the first text is used to adjust the first image; receiving multiple preview images from the cloud, where the multiple preview images are generated based on the first image and the first text; sending operation information input by the user to the cloud, where the operation information is used to retrieve a second image associated with a target image, where the target image is included in the multiple preview images; and receiving the second image from the cloud.

[0033] In this application, the user can input the first picture and the first text on the client, and use the first text to guide the cloud to adjust the first picture, thereby obtaining multiple preview pictures. The user can select a suitable target picture from the preview pictures, and the cloud will search based on the target picture, which can narrow the search scope and improve the accuracy of image retrieval.

[0034] In one possible implementation, the method is applied to a response system based on a large language model (LLM); the operation information includes a target image, or the operation information includes a target image and a second text.

[0035] In this possible implementation, if the preview image contains a target image that the user is satisfied with, image retrieval can be performed based on the target image. Of course, retrieval can also be performed based on the target image and the extended text corresponding to the target image. If the preview image does not contain an image that the user is satisfied with, the target image can be used as the input image for the second image editing, and the second text can guide the cloud to edit the target image, and further provide the user with a second wave of preview images until the user selects a satisfactory image. From this possible implementation, it can be seen that in the response system, different models can be called through LLM to implement different functions such as image retrieval or image editing, thereby enriching the functions of the response system.

[0036] In a possible implementation, if the operation information includes a target image, the second image is retrieved based on the extended text of the first text and the target image by calling the image editing retrieval model through the LLM.

[0037] In this possible implementation, the image editing retrieval model is based on the extended text corresponding to the target image and the target retrieval of the second image, which can improve the accuracy of the retrieved second image.

[0038] In one possible implementation, if the second text indicates that the target image meets the user's expectations, the second image is retrieved by calling the image editing retrieval model through the LLM based on the extended text corresponding to the target image, the second text, and the target image.

[0039] In this possible implementation, the image editing retrieval model retrieves the second image based on the extended text corresponding to the target image, the second text, and the target image, which can improve the accuracy of the retrieved second image.

[0040] In one possible implementation, if the second text indicates that the target image does not meet the user's expectations, the method also includes: receiving multiple preview images associated with the target image and the second text from the cloud, the multiple preview images associated with the target image and the second text are obtained by LLM calling the image editing generation model, the image editing generation model obtains the multiple preview images associated with the target image and the second text based on multiple extended texts of the second text and the target image; sending a third image, or a third image and a third text to the cloud; the third image, or the third image and the third text is used to determine the second image, and the third image is included in the multiple preview images associated with the target image and the second text.

[0041] In this possible implementation, the image editing generation model refers to a model that uses the extended text of the second text and the target image to further generate the next wave of preview images. The generation process can be understood by referring to the process of generating preview images from the first text and the first image. This multiple rounds of preview image generation can ensure that users obtain satisfactory preview images, thereby improving the accuracy of the retrieved second image.

[0042] In a possible implementation, when receiving the second image from the cloud, the method further includes receiving link information associated with the second image from the cloud.

[0043] In this possible implementation, the link information associated with the second picture may be description information of the product in the second picture and a purchase link, etc. In this way, the efficiency of communication with the user can be improved.

[0044] In a third aspect of the present application, a cloud device is provided for executing the method of the first aspect or any possible implementation of the first aspect. Specifically, the cloud device includes modules or units for executing the method of the first aspect or any possible implementation of the first aspect, such as a processing unit and a sending unit.

[0045] In a fourth aspect, the present application provides a client for executing the method of the second aspect or any possible implementation of the second aspect. Specifically, the client includes modules or units for executing the method of the second aspect or any possible implementation of the second aspect, such as a sending unit and a receiving unit.

[0046] In the fifth aspect of the present application, a cloud device is provided, comprising a transceiver, a processor and a memory, wherein the transceiver and the processor are coupled to the memory, and the memory is used to store programs or instructions. When the programs or instructions are executed by the processor, the client executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0047] In a sixth aspect of the present application, a client is provided. The client may include at least one processor, a memory, and a communication interface. The processor is coupled to the memory and the communication interface. The memory is configured to store instructions, the processor is configured to execute the instructions, and the communication interface is configured to communicate with other network elements under the control of the processor. When executed by the processor, the instructions cause the processor to perform the method of the second aspect or any possible implementation of the second aspect.

[0048] The seventh aspect of the present application provides a chip system, which includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive signals from the memory of the cloud device and send signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the cloud device executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0049] In an eighth aspect of the present application, a chip system is provided, which includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the client's memory and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the client executes the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0050] The ninth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction runs on a computer device, the computer device executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0051] The tenth aspect of the present application provides a computer-readable storage medium having a computer program or instruction stored thereon. When the computer program or instruction runs on a computer device, the computer device executes the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0052] In an eleventh aspect, the present application provides a computer device program product, which includes a computer device program code. When the computer device program code is executed on a computer device, the computer device executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0053] The twelfth aspect of the present application provides a computer device program product, which includes a computer device program code. When the computer device program code is executed on a computer device, the computer device executes the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0054] The thirteenth aspect of the present application provides a computer device cluster, comprising at least one computer device, each computer device comprising a processor and a memory; the processor of at least one computer device is used to execute instructions stored in the memory of at least one computer device, so that the computer device cluster executes the method in the aforementioned first aspect or any possible implementation of the first aspect.

[0055] In the fourteenth aspect, the present application provides a retrieval system, which includes a client and a cloud device. The cloud device is used to execute the method in the aforementioned first aspect or any possible implementation of the first aspect, and the client is used to execute the method in the aforementioned second aspect or any possible implementation of the second aspect.

[0056] Among them, the technical effects brought about by the third to fourteenth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or different possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] FIG1A is a schematic diagram of an architecture of a retrieval system provided in an embodiment of the present application;

[0058] FIG1B is a schematic diagram of image retrieval in a shopping scenario provided by an embodiment of the present application;

[0059] FIG1C is a schematic diagram of an architecture of a cloud system provided in an embodiment of the present application;

[0060] FIG2A is a schematic structural diagram of a terminal device provided in an embodiment of the present application;

[0061] FIG2B is a schematic diagram of a structure of a cloud device provided in an embodiment of the present application;

[0062] FIG3 is a schematic diagram of an embodiment of a method for image retrieval provided in an embodiment of the present application;

[0063] FIG4 is a schematic diagram of an example of an input interface of a client in an embodiment of the present application;

[0064] FIG5 is a schematic diagram of an example scenario of generating a preview image in an embodiment of the present application;

[0065] FIG6 is a schematic diagram showing the relationship between a large language model and a text analysis model according to an embodiment of the present application;

[0066] FIG7 is a schematic diagram of another embodiment of image retrieval in the embodiment of the present application;

[0067] FIG8 is a schematic diagram of an architecture of a server provided in an embodiment of the present application;

[0068] FIG9 is a schematic diagram of an architecture of a retrieval system according to an embodiment of the present application;

[0069] FIG10 is a schematic structural diagram of a cloud device provided in an embodiment of the present application;

[0070] FIG11 is a schematic diagram of the structure of a client provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] The following describes the embodiments of the present application in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present application, rather than all the embodiments. Those skilled in the art will appreciate that with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] The terms "first," "second," and the like in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0073] The present application provides an image retrieval method for improving the accuracy of image retrieval. The present application also provides corresponding devices, systems, computer-readable storage media, and computer program products. These are described in detail below.

[0074] To facilitate understanding, the following briefly introduces the technical terms involved in the embodiments of this application:

[0075] 1. Text retrieval (Query): The user enters text in the search box and selects the document candidate set or image candidate set that is most relevant to the text from a large-scale web document library (the document library can be a library of hundreds of billions of documents).

[0076] 2. Image retrieval: Searches based on the image input by the user and returns images similar to the input image; also known as image search, it can be understood as searching for images by image.

[0077] 3. Image editing retrieval: Image editing retrieval allows users to input images and text at the same time. The text can be used to edit and modify local information in the image. The images returned by the retrieval meet the content of the text and are as similar to the input image as possible.

[0078] 4. Image editing generation: Image editing generation allows users to input images and text at the same time. The text can be used to edit and modify local information in the image. A new image is generated based on the image and text. The new image meets the text requirements, and other information remains consistent with the input image.

[0079] 5. Answering system: Generally refers to a system that searches for relevant documents and then answers user questions based on the understanding of the document information.

[0080] 6. Conversational search and response system: This system adds multi-round conversation capabilities to the search and response system, can understand the context, and thus implement multi-round continuous response functions.

[0081] 7. Large Language Model (LLM): After training, LLM internalizes knowledge into model parameters, allowing it to directly generate answers to user queries. LLMs can leverage image retrieval models, image editing retrieval models, or image editing generation models to implement different functions.

[0082] FIG1A is a schematic diagram of an architecture of a search provided by an embodiment of the present application.

[0083] As shown in FIG1A , the retrieval system provided in an embodiment of the present application includes a cloud and multiple clients, and the cloud can communicate with the multiple clients via a network. The cloud can be software or services of a cloud platform, or software or services deployed on nodes in a network, such as edge nodes. The cloud can run on an independent physical machine or on virtualized resources. The client can be a terminal device or an application, for example, the application runs on a terminal device for user use. The application can be a search engine, an answering application, or a shopping application, etc.

[0084] The retrieval system may be a browser search system, a search response system, or a shopping platform system, etc.

[0085] The client can send a query request to the cloud, the query request including a first text and a first image, where the first text is used to adjust the first image. A cloud-side device in the cloud can generate multiple preview images based on the first image and the first text received from the client, and then send the multiple preview images to the client. Then, based on the operation information received from the client, it can retrieve a second image associated with a target image, where the target image is included in the multiple preview images, and then send the second image to the client.

[0086] Taking a shopping scenario as an example, referring to FIG1B , the first image input by the user may be a "black short-sleeved shirt" image 101, and the first text may be "A floral pattern would be even better" 102. Based on the "black short-sleeved shirt" image 101 and the text "A floral pattern would be even better" 102, the cloud can generate four preview images, as shown in FIG1B . After the client receives these four preview images, the user can select one of the four preview images as the target image. For example, in FIG1B , the user selects "a black short-sleeved shirt with evenly distributed flowers" 103 as the target image. The cloud can retrieve a suitable second image based on "a black short-sleeved shirt with evenly distributed flowers" 103. As shown in FIG1B , the cloud returns two second images to the client, namely, images 104 and 105. The flower distribution on the short-sleeved shirts in images 104 and 105 is highly similar to that in image 103. Image 104 shows a short-sleeved shirt with evenly distributed black flowers, while image 105 shows a short-sleeved shirt with evenly distributed white flowers. The user can select the product they like by comparing the colors and making a purchase. The cloud can also provide users with more color options, and these short-sleeved shirts with flowers in different colors can be presented in one picture.

[0087] In this application, the user can input the first picture and the first text on the client, and use the first text to guide the cloud to adjust the first picture, thereby obtaining multiple preview pictures. The user can select a suitable target picture from the preview pictures, and the cloud will search based on the target picture, which can narrow the search scope and improve the accuracy of image retrieval.

[0088] The cloud-side device in the cloud in Figure 1A can be a working node or a scheduling node in the cloud system. After the scheduling node receives a query request from the client, the scheduling node can execute the corresponding retrieval process. The scheduling node can also assign the query request to one or more working nodes in the cloud system, and one or more working nodes can execute the corresponding retrieval process.

[0089] The function of the scheduling node can be implemented by software or hardware.

[0090] As an example of a software functional unit, a scheduling node may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, a scheduling node may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0091] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Inter-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0092] As an example of a hardware functional unit, a scheduling node may include at least one computing device, such as a server. Alternatively, the scheduling node may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0093] The multiple computing devices included in a scheduling node can be distributed in the same zone or in different zones. The multiple computing devices included in a scheduling node can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in a scheduling node can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, and other computing devices.

[0094] A worker node can be a physical machine, a virtual machine (VM), or a container. It can include one or more central processing units (CPUs) and graphics processing units (GPUs). A worker node can also be a CPU or a GPU.

[0095] The client can be a terminal device. A terminal device, also known as user equipment (UE), mobile station (MS), mobile terminal (MT), etc., is a device that includes wireless communication capabilities (providing voice / data connectivity to users), such as a handheld device with wireless connection capabilities. Currently, some examples of terminal devices include: mobile phones, tablets, laptops, PDAs, notebook computers, wireless routers, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving cars, wireless terminals in the Internet of Vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes. For example, a wireless terminal in the Internet of Vehicles can be an onboard device, a complete vehicle device, an onboard module, a vehicle, etc. Wireless terminals in industrial control can be robots, for example. For example, wireless terminals in unmanned driving can be drones. These terminal devices can run Android, iOS, Windows, or other operating systems. These terminal devices can run applications that require rendering of application scenes to generate two-dimensional images, such as gaming applications, lock screen applications, or map applications.

[0096] The cloud in Figure 1A above may be located in a cloud system. The architecture of the cloud system can be understood with reference to Figure 1C . As shown in Figure 1C , the cloud system includes a cloud platform and basic resources. The cloud platform includes a cloud platform manager. The scheduling node described above may be the cloud platform manager in Figure 1C . Basic resources may include multiple servers, each of which may be a worker node, or each server may include multiple worker nodes.

[0097] The working node in Figure 1C can be a computing device card or a virtual machine (VM). The computing device card can be at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a neural network processing unit (NPU).

[0098] The cloud platform manager maintains or regularly collects information about each worker node in the underlying resources, such as resource usage (resource utilization rate or resource idle rate) on each worker node. This information can be used as auxiliary decision-making information when allocating query requests.

[0099] The cloud platform manager can receive query requests from the client, and then the cloud platform manager can execute the corresponding retrieval process. The cloud platform manager can also assign the query request to one or more working nodes in the cloud system, and the one or more working nodes can execute the corresponding retrieval process.

[0100] After the working node completes the retrieval, the cloud platform manager can return the preview image and the second image to the client.

[0101] The above-mentioned client takes a terminal device as an example. The structure of the terminal device provided in the embodiment of the present application can be understood by referring to FIG. 2A below, and the structure of the cloud device in the cloud can be understood by referring to FIG. 2B below.

[0102] Please refer to Figure 2A, which is a structural diagram of a terminal device provided in an embodiment of the present application. As shown in Figure 2A, the terminal device may include a processor 101, a transceiver 102, a memory 103, a display 104 and a bus 105. The processor 101, the transceiver 102, the memory 103 and the display 104 are interconnected via the bus 105. In an embodiment of the present application, the processor 101 is used to control and manage the actions of the terminal device 10, for example, the processor 101 is used to respond to the process of the user inputting a query request. The transceiver 102 is used to support the communication of the terminal device 10, for example: the transceiver 102 can execute the steps of sending a query request and receiving a preview image and a second image. The memory 103 is used to store the program code and data of the terminal device 10, and the display 104 is used to display the preview image and the second image.

[0103] The processor 101 may be a central processing unit (CPU), a general-purpose processor (GPPC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PLD), a transistor logic device (TLD), a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The bus 105 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG. 2A shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0104] FIG2A above introduces the structure of the terminal device. The structure of the cloud device will be introduced below in conjunction with FIG2B .

[0105] Figure 2B is a schematic diagram of a possible logical structure of a cloud device provided in an embodiment of the present application. As shown in Figure 2B, the cloud device 20 provided in an embodiment of the present application includes: a processor 201, a communication interface 202, a memory 203 and a bus 204. The processor 201, the communication interface 202 and the memory 203 are interconnected via the bus 204. In an embodiment of the present application, the processor 201 is used to control and manage the actions of the cloud device 20. For example, the processor 201 is used to generate multiple preview images based on the first image and the first text from the client. The communication interface 202 is used to support the cloud device 20 to communicate. For example, the communication interface 202 can receive the first image and the first text, and send the preview image and the second image. The memory 203 is used to store the program code and data of the cloud device 20.

[0106] The processor 201 may be a central processing unit (CPU), a general-purpose processor (GPOR), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device (PLD), a transistor logic device (TLD), a hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. A processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The bus 204 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, for example. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG. 2B shows only one thick line, but this does not imply that there is only one bus or only one type of bus.

[0107] The following describes the image retrieval method provided by the embodiment of the present application. The content of the method involving cloud execution can be executed by the cloud, or by a component of the cloud (such as a processor, chip, or chip system, etc.). The content involving client execution can be executed by the client, or by a component of the client (such as a processor, chip, or chip system, etc.).

[0108] FIG3 is a schematic diagram of an embodiment of a method for image retrieval provided in an embodiment of the present application.

[0109] 301. The client sends a first image and a first text input by the user to the cloud. Correspondingly, the cloud receives the first image and the first text from the client.

[0110] In this application, the first image and the first text may be sent through one query request or may be sent separately.

[0111] The first text is used to adjust the first picture.

[0112] The interface for inputting the first picture and the first text on the client can be understood by referring to Figure 4. As shown in Figure 4, the first text can be entered in the text input box 401, the first picture can be selected through the picture option box 402, and then the first picture and the first text can be sent by clicking the send button 403.

[0113] 302. The cloud generates multiple preview images based on the first image and the first text from the client.

[0114] The process of step 302 may be: expanding the first text in the cloud to obtain multiple extended texts; and then generating a preview image corresponding to each extended text according to each extended text and the first image.

[0115] The cloud can expand around the content of the first text. For example, if the first text is a flower pattern configured on a first picture, the text can be expanded around the type of flower, the position of the flower on the first picture, etc.

[0116] The extended text can be generated by the cloud based on the first text, or can be text found by the cloud in a text design database whose similarity to the first text meets the first condition. Alternatively, the extended text can be found by the cloud in a text design database whose similarity to the first text and design popularity meet the second condition.

[0117] The text design database may be pre-built and continuously updated. The texts in the text design database may be collected based on historical text search records. Of course, the texts in the text design database may also be obtained through other means.

[0118] In this application, the first text can be used to search for texts with high similarity to the first text in the text design database. The first condition can be a threshold value. Taking the similarity value in the range of 0 to 1 as an example, the threshold value can be 0.5 or other values. Taking 0.5 as an example, texts with a similarity greater than 0.5 can be understood as texts whose similarity to the first text meets the first condition. The first condition can also be a restriction on the number of texts, such as: the top N texts with the highest similarity to the first text are texts whose similarity to the first text meets the first condition. Wherein, N is an integer greater than 1.

[0119] In this application, a first text can be used to search for texts with high similarity to the first text in a text design database. The popularity of these texts with high similarity can then be determined. The similarity and popularity can then be combined to determine the extended text. Each text in the text design database can have its popularity calculated in advance based on the number of times it has been used. The popularity can be a value between 0 and 1. The second condition can include one or two thresholds. If there is only one threshold, the similarity and popularity of the texts can be weighted to obtain a weighted value (the weight of similarity is usually greater than the popularity). If the weighted value is greater than the threshold, it means that the text meets the second condition and is an extended text of the first text. If the second condition includes two thresholds, one threshold can be a similarity threshold, such as 0.5; the other threshold can be a threshold for the number of extended texts, such as N, where N is a positive integer. If the similarity of the found text to the first text is greater than 0.5, the extended texts with a similarity greater than 0.5 can be sorted according to their popularity, and then the N extended texts with the highest popularity are selected as the final selected extended texts.

[0120] The above process of generating a preview image through a text design database can also be understood by referring to FIG. 5 .

[0121] As shown in Figure 5, the cloud can compare the similarity of the first text "It would be better with a flower pattern" with the text in the text design database. If the first condition requires that the similarity between the extended text and the first text reaches 0.7, the cloud can match four extended texts shown in Figure 5 from the text design database, namely "multiple small flowers in the center", "small flower pattern in the upper left corner", "a red rose in the center", and "multiple blooming flowers".

[0122] The similarities between these four extended texts and the first text can be understood by referring to Table 1, as shown in Table 1:

[0123] Table 1: Similarity between the extended text and the first text

[0124] After the cloud determines the extended text, it can use a picture editing generation model based on each extended text and the first picture, input the first picture and each extended text into the picture editing generation model, generate different design schemes, and then generate preview pictures based on the different design schemes.

[0125] In addition, the cloud can also determine the extended text based on the similarity and popularity of the text. For example, in Figure 5, multiple texts with high similarity to the first text "It would be better if there were flower patterns" can be found from the text design database according to the similarity requirement. Then, the extended text can be determined by further combining the popularity of each text with high similarity. For example, texts with a similarity higher than 0.7 are sorted in descending order of popularity, and then the texts ranked in the top N (N=4) are selected as extended texts, and the four extended texts in Table 2 can be obtained.

[0126] Table 2: Similarity between the extended text and the first text, and the popularity of the extended text

[0127] After obtaining the extended text, a preview image can be generated by inputting the extended text and the first image into the image editing generation model.

[0128] In an embodiment of the present application, by expanding the text, the preview images obtained can provide preview images that meet the user's requirements in the first text from multiple angles, thereby increasing the richness of the preview images within the scope of the user's requirements.

[0129] 303. The cloud sends multiple preview images to the client. Correspondingly, the client receives multiple preview images from the cloud.

[0130] 304. The client sends the user input operation information to the cloud. Correspondingly, the cloud receives the operation information from the client.

[0131] The operation information is used to retrieve a second image associated with a target image, and the target image is included in multiple preview images.

[0132] In an embodiment of the present application, the operation information includes a target image, or the operation information includes a target image and a second text.

[0133] In a response system based on a large language model (LLM), if the operation information includes a target image but not a second text, the LLM can call an image editing retrieval model to retrieve the second image based on the target image and the extended text corresponding to the target image.

[0134] The response system may also include a text analysis model. The LLM may call the text analysis model to analyze the second text. Then, based on the analysis result of the second text, the LLM may decide to call the image editing retrieval model or the image editing generation model.

[0135] The structure of the text analysis model and its relationship with the LLM can be understood by referring to Figure 6. As shown in Figure 6, the text analysis model can include a sentiment analysis model and an entity extraction model. The sentiment analysis model analyzes the user's tone and emotions in the text, while the entity extraction model analyzes entities within the text. Entities are objects within the text. After analyzing the second text, the text analysis model returns the analysis results to the LLM.

[0136] 305. The cloud retrieves a second image associated with the target image based on the operation information from the client, where the target image is included in the plurality of preview images.

[0137] In step 305 , if the operation information does not include the second text, the cloud can call the image editing retrieval model through the LLM to retrieve the associated second image based on the extended text corresponding to the target image and the target image.

[0138] If the operation information includes a second text, the process can be understood by referring to Figure 7. The text analysis model 701 analyzes the second text to determine whether the target image meets the user's expectations. If the second text is a sentence such as "pretty good", "not bad", "it's what I want", etc., which indicates user satisfaction, it means that the target image meets the user's expectations. Then LLM702 calls the picture editing retrieval model 703, and the picture editing retrieval model 703 retrieves the second image based on the extended text corresponding to the target image and the target image.

[0139] If the second text contains statements such as "It's not what I want" or "The flowers are too scattered," indicating user dissatisfaction, the target image does not meet the user's expectations. LLM 702 then invokes image editing generation model 704, which generates multiple preview images associated with the target image and the second text. The process for generating multiple preview images associated with the target image and the second text can be understood by referring to the process of generating preview images for the first text and the first image. The cloud then sends this wave of preview images to the client. If the user selects a satisfactory target image from this wave of preview images, the cloud can retrieve a second image based on the target image that the user is satisfied with. If the user is still not satisfied, the above process of generating preview images can be repeated until a satisfactory target image is obtained.

[0140] The image editing retrieval model can retrieve the second image from the image vector database. The image vector database can be pre-built and continuously updated. The images in the image vector database can be collected from the Internet and can be continuously updated. Of course, the second image can also be searched from the Internet. In this application, by retrieving the second image from the image vector database, the speed of second image retrieval can be improved.

[0141] 306. The cloud sends the second image to the client. Correspondingly, the client receives the second image from the cloud.

[0142] In the embodiment of the present application, when the cloud sends the second picture, it can also send link information associated with the second picture. Correspondingly, when the client receives the second picture, it also receives link information associated with the second picture from the cloud.

[0143] The link information associated with the second picture may be description information of the product in the second picture and a purchase link, etc. In this way, the communication efficiency with the user can be improved.

[0144] In this application, the user can input the first picture and the first text on the client, and use the first text to guide the cloud to adjust the first picture, thereby obtaining multiple preview pictures. The user can select a suitable target picture from the preview pictures, and the cloud will search based on the target picture, which can narrow the search scope and improve the accuracy of image retrieval.

[0145] In the embodiment of the present application, the execution process in the cloud can be configured in the platform software for machine learning and deep learning, and can be program code deployed on the server hardware. Taking the architecture shown in Figure 8 as an example, the program code in the cloud of the present application can exist inside the runtime engine, memory management and communication management modules of the above-mentioned platform software, as well as outside the existing modules. During operation, the program code of the present application can run in the host memory and / or graphics processing unit (GPU) memory of the server. The architecture shown in Figure 8 is the implementation form in the server and platform software in the embodiment of the present application, wherein the hardware layer includes a GPU and a memory. In addition, the memory will store a text design database and an image vector database. The software layer includes an LLM, an image editing generation model and an image editing retrieval model. Of course, the software layer can also include a text analysis model. The calling relationship between the LLM, the image editing generation model, the image editing retrieval model and the text analysis model can be understood by referring to the previous introduction. When the server executes the process of the present application, it will call the text design database in the memory through the GPU to generate the extended text, and call the image in the image vector database to obtain the preview image. These processes can be understood by referring to the previous introduction.

[0146] The text design database and the image vector database in the embodiment of the present application can be obtained through offline training and can be updated online. As shown in Figure 9, the retrieval system is roughly divided into an offline system, an online system and a storage module. The main task of the offline system is to construct design texts and product image feature vectors based on Internet product images and texts, and to screen the design texts and product images. The process may include: deduplication of product images, and generating image representation vectors based on the image pre-training model, constructing a product image index, and constructing a picture vector database. Based on the product text, the product attributes are classified and the specific content is extracted; after completion, deduplication and text cleaning are performed to construct a text design database.

[0147] In an embodiment of the present application, the offline system can also retrain the image editing generation model and the image editing retrieval model based on product images and text.

[0148] The online system's primary task is to generate preview images and retrieve products based on user requests, using an editable image generation model and an editable image retrieval model. The storage module primarily stores design text, product image representation vectors, the LLM large model, the editable image retrieval model, and the editable image generation model for easy access by both the offline and online systems.

[0149] After receiving the user input request, the online system passes it to the LLM for decision-making; the LLM analyzes the user input and decides whether to start the image editing retrieval process. If so, it calls the image editing generation module.

[0150] The image editing generation module expands the text input by the user to generate multiple extended texts, and generates corresponding preview images based on the image editing generation model and returns them to the LLM.

[0151] LLM responds to the user's request based on the generated preview image; the user provides feedback on the generated preview image.

[0152] LLM passes user feedback to the text analysis model, which obtains key information from the user feedback and returns the analysis results to LLM.

[0153] Based on the analysis results returned by the text analysis module, LLM decides whether to continue calling the image editing generation module or the image editing retrieval module; if the user is satisfied, LLM calls the image editing retrieval module; if the user is not satisfied, LLM calls the image editing generation module to continue generating the next wave of preview images.

[0154] Once the user is satisfied with the generated preview image, the image editing retrieval model performs image editing retrieval with the user-selected target image and corresponding text as input, and returns the retrieved image to the LLM.

[0155] LLM combines the retrieved images and responds to the user.

[0156] After the program code of the embodiment of the present application is deployed in the cloud, it can provide external services through a remote link, such as: a user downloads an application (APP), and the user can interact with the cloud through the APP to obtain a satisfactory picture.

[0157] The solution provided by the embodiment of this application combines preview image generation with product retrieval: based on the image and text entered by the user, a preview is generated using image editing technology, allowing the user to obtain satisfactory images. In addition, this application recommends expansion solutions to users through text expansion, achieving diversified recommendations, while also helping to collect the user's editing intent, solving the problem of short user-entered text and a large search range. In addition, retrieval can be based on images that the user is satisfied with, achieving accurate retrieval of images.

[0158] The above describes the image retrieval system and the image retrieval method. The following describes the related devices with reference to the accompanying drawings.

[0159] As shown in FIG10 , a structure of a cloud device 100 provided in an embodiment of the present application includes:

[0160] The processing unit 1001 is configured to generate a plurality of preview images according to a first image and a first text from a client, wherein the first text is used to adjust the first image.

[0161] The sending unit 1002 is configured to send multiple preview images to the client.

[0162] The processing unit 1001 is further configured to retrieve a second image associated with the target image according to operation information from the client, where the target image is included in the plurality of preview images.

[0163] The sending unit 1002 is further configured to send the second image to the client.

[0164] In an embodiment of the present application, the user can input the first picture and the first text on the client, and use the first text to guide the cloud to adjust the first picture, thereby obtaining multiple preview pictures. The user can select a suitable target picture from the preview pictures, and the cloud will search based on the target picture, which can narrow the search scope and improve the accuracy of the picture retrieval.

[0165] Optionally, the processing unit 1001 is configured to expand the first text to obtain multiple extended texts; and generate a preview image corresponding to each extended text according to each extended text and the first image.

[0166] Optionally, the multiple extended texts are texts in a text design database whose similarity with the first text meets a first condition.

[0167] Optionally, the multiple extended texts are texts in a text design database whose similarity and design popularity with the first text meet the second condition.

[0168] Optionally, the second picture is a picture retrieved from a picture vector database.

[0169] Optionally, the cloud device 100 is applied to a response system based on a large language model (LLM); the operation information includes a target image, or the operation information includes a target image and a second text.

[0170] Optionally, if the operation information includes a target image, the processing unit 1001 is configured to call the image editing retrieval model through the LLM, and retrieve a second image associated with the target image based on the extended text corresponding to the target image and the target image.

[0171] Optionally, if the second text indicates that the target image meets the user's expectations, the processing unit 1001 is used to call the image editing retrieval model through the LLM to retrieve the associated second image based on the extended text corresponding to the target image, the second text and the target image.

[0172] Optionally, if the second text indicates that the target image does not meet user expectations, the processing unit 1001 is used to call the picture editing generation model through LLM to generate multiple preview images associated with the target image and the second text. The picture editing generation model is used to obtain multiple preview images associated with the target image and the second text based on multiple extended texts of the second text and the target image to determine the second image.

[0173] Optionally, the LLM-based answering system further includes a text analysis model, which is used to analyze the second text to determine whether the target image meets user expectations.

[0174] Optionally, the sending unit 1002 is further configured to send link information associated with the second image to the client.

[0175] In the embodiment of the present application, the operations performed by each unit in the cloud 100 are similar to those described in the embodiments shown in Figures 3 to 9 above, and will not be repeated here.

[0176] As shown in FIG11 , the embodiment of the present application further provides a structure of a client 110 including:

[0177] The sending unit 1101 is configured to send a first image and a first text input by a user to the cloud, where the first text is used to adjust the first image.

[0178] The receiving unit 1102 is configured to receive a plurality of preview images from the cloud, where the plurality of preview images are generated based on the first image and the first text.

[0179] The sending unit 1101 is used to send operation information input by the user to the cloud, where the operation information is used to retrieve a second image associated with a target image, where the target image is included in multiple preview images.

[0180] The receiving unit 1102 is configured to receive a second image from the cloud.

[0181] Optionally, the client 110 is applied to a response system based on a large language model (LLM); the operation information includes a target image, or the operation information includes a target image and a second text.

[0182] Optionally, if the operation information includes a target image, the second image is retrieved based on the extended text of the target image and the target image by calling the image editing retrieval model through the LLM.

[0183] Optionally, if the second text indicates that the target image meets the user's expectations, the second image is retrieved based on the extended text of the target image, the second text, and the target image by calling the image editing retrieval model through the LLM.

[0184] Optionally, if the second text indicates that the target image does not meet the user's expectations, the receiving unit 1102 is further used to receive multiple preview images associated with the target image and the second text from the cloud. The multiple preview images associated with the target image and the second text are obtained by LLM calling the image editing generation model. The image editing generation model is used to obtain multiple preview images associated with the target image and the second text based on multiple extended texts of the second text and the target image.

[0185] The sending unit 1101 is also used to send a third picture, or a third picture and a third text to the cloud. The third picture, or the third picture and the third text are used to determine the second picture, and the third picture is included in the multiple preview pictures associated with the target picture and the second text.

[0186] Optionally, the receiving unit 1102 is further configured to receive link information associated with the second image from the cloud.

[0187] In the embodiment of the present application, the operations performed by each unit in the client 110 are similar to those described in the embodiments shown in Figures 3 to 9 above, and will not be repeated here.

[0188] In another embodiment of the present application, a computer-readable storage medium is also provided, in which computer-executable instructions are stored. When the processor of the cloud device executes the computer-executable instructions, the cloud device executes the steps executed by the cloud in Figures 3 to 9 above.

[0189] In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor of the client executes the computer-executable instructions, the client executes the steps performed by the client in Figures 3 to 9 above.

[0190] In another embodiment of the present application, a computer program product is further provided. The computer program product includes computer program code. When the computer program code is executed on a computer, the computer device executes the steps executed by the cloud or client in Figures 3 to 9 above.

[0191] In another embodiment of the present application, a chip system is also provided, which includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive signals from the memory of the terminal and send signals to the processor, and the signals include computer instructions stored in the memory; when the processor executes the computer instructions, the terminal executes the steps performed by the cloud device or client in Figures 3 to 9 above. In one possible design, the chip system may also include a memory, which is used to store program instructions and data necessary for the control device. The chip system can be composed of chips, or it can include chips and other discrete devices.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0193] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0194] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in whole or in part through software, hardware, firmware, or any combination thereof.

[0195] When software is used to implement the integrated unit, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrations. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

Claims

1. A method for image retrieval, characterized in that: include: generating a plurality of preview images according to a first image and a first text from a client, wherein the first text is used to adjust the first image; Sending the plurality of preview images to the client; Retrieving a second picture associated with a target picture according to the operation information from the client, wherein the target picture is included in the plurality of preview pictures; Send the second picture to the client.

2. The method according to claim 1, characterized in that The step of generating a plurality of preview images according to the first image and the first text from the client includes: Expanding the first text to obtain a plurality of expanded texts; A preview image corresponding to each extended text is generated according to each extended text and the first image.

3. The method according to claim 2, characterized in that The multiple extended texts are texts in a text design database whose similarity with the first text meets a first condition.

4. The method according to claim 2, characterized in that: The multiple extended texts are texts in the text design database whose similarity and design popularity with the first text meet the second condition.

5. The method according to any one of claims 1 to 4, characterized in that: The second picture is a picture retrieved from a picture vector database.

6. The method according to any one of claims 1 to 5, characterized in that: The method is applied to a response system based on a large language model (LLM); The operation information includes the target image, or the operation information includes the target image and a second text.

7. The method according to claim 6, characterized in that If the operation information includes the target image, correspondingly, the retrieving a second image associated with the target image includes: The picture editing retrieval model is called by the LLM to retrieve a second picture associated with the target picture based on the extended text corresponding to the target picture and the target picture.

8. The method according to claim 6, characterized in that If the second text indicates that the target image meets the user's expectations, correspondingly, retrieving a second image associated with the target image includes: The picture editing retrieval model is called by the LLM to retrieve the associated second picture based on the extended text corresponding to the target picture, the second text and the target picture.

9. The method according to claim 6, characterized in that If the second text indicates that the target image does not meet the user's expectations, correspondingly, retrieving a second image associated with the target image includes: The LLM calls a picture editing generation model to generate multiple preview pictures associated with the target picture and the second text. The picture editing generation model is used to obtain multiple preview pictures associated with the target picture and the second text based on multiple extended texts of the second text and the target picture to determine the second picture.

10. The method according to any one of claims 6 to 9, characterized in that: The LLM-based answering system also includes a text analysis model, which is used to analyze the second text to determine whether the target image meets user expectations.

11. The method according to any one of claims 1 to 10, characterized in that: When sending the second picture to the client, the method further includes: Send link information associated with the second image to the client.

12. A method for image retrieval, characterized in that: include: Sending a first picture and a first text input by a user to a cloud, where the first text is used to adjust the first picture; Receiving a plurality of preview images from the cloud, where the plurality of preview images are generated based on the first image and the first text; Sending operation information input by the user to the cloud, wherein the operation information is used to retrieve a second image associated with a target image, wherein the target image is included in the plurality of preview images; Receive the second picture from the cloud.

13. The method according to claim 12, characterized in that The method is applied to a response system based on a large language model (LLM); The operation information includes the target image, or the operation information includes the target image and a second text.

14. The method according to claim 13, characterized in that If the operation information includes the target image, the second image is retrieved based on the extended text of the target image and the target image by calling the image editing retrieval model through the LLM.

15. The method according to claim 13, characterized in that If the second text indicates that the target image meets the user's expectations, the second image is retrieved by the LLM calling the image editing retrieval model based on the extended text of the target image, the second text and the target image.

16. The method according to claim 13, characterized in that If the second text indicates that the target image does not meet the user's expectations, the method further includes: Receiving multiple preview images associated with the target image and the second text from the cloud, wherein the multiple preview images associated with the target image and the second text are obtained by the LLM calling a picture editing generation model, and the picture editing generation model is used to obtain the multiple preview images associated with the target image and the second text based on multiple extended texts of the second text and the target image; A third image, or a third image and a third text are sent to the cloud; the third image, or the third image and the third text are used to determine the second image, and the third image is included in a plurality of preview images associated with the target image and the second text.

17. The method according to any one of claims 12 to 16, characterized in that: When receiving the second picture from the cloud, the method further includes: Receive link information associated with the second image from the cloud.

18. A cloud device, characterized in that: include: A processing unit, configured to generate a plurality of preview images according to a first image and a first text from a client, wherein the first text is used to adjust the first image; A sending unit, used for sending a plurality of preview images generated by the processing unit to the client; The processing unit is further configured to retrieve a second image associated with a target image according to the operation information from the client, wherein the target image is included in the plurality of preview images; The sending unit is further used to send the second picture retrieved by the processing unit to the client.

19. A client, characterized in that: include: A sending unit, configured to send a first image and a first text input by a user to a cloud, wherein the first text is used to adjust the first image; A receiving unit, configured to receive a plurality of preview images from the cloud, where the plurality of preview images are generated based on the first image and the first text; The sending unit is further used to send operation information input by the user to the cloud, where the operation information is used to retrieve a second image associated with a target image, where the target image is included in the plurality of preview images; The receiving unit is further configured to receive the second image from the cloud.

20. A cloud device, characterized in that: include: A communication interface, a processor and a memory, wherein the communication interface and the processor are coupled to the memory, and the memory is used to store programs or instructions. When the programs or instructions are executed by the processor, the cloud device executes the method as described in any one of claims 1-11.

21. A client, characterized in that: include: A transceiver, a processor and a memory, wherein the transceiver and the processor are coupled to the memory, and the memory is used to store programs or instructions. When the programs or instructions are executed by the processor, the client executes the method according to any one of claims 12 to 17.

22. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computer device, the computer device executes the method according to any one of claims 1 to 17.

23. A computer program product, characterized in that The computer program product comprises a computer program code, and when the computer program code is run on a computer device, the computer device is caused to perform the method according to any one of claims 1 to 17.

24. A chip system, characterized in that: The chip system includes one or more interface circuits and one or more processors; the interface circuit and the processor are interconnected by lines; the interface circuit is used to receive signals from the memory of the cloud device and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the cloud device executes the method as described in any one of claims 1-11; or, the interface circuit is used to receive signals from the memory of the client and send signals to the processor, the signals including computer instructions stored in the memory; when the processor executes the computer instructions, the client executes the method as described in any one of claims 12-17.

25. A computer device cluster, characterized in that: comprising at least one computer device, each computer device comprising a processor and a memory; The processor of the at least one computer device is used to execute instructions stored in the memory of the at least one computer device, so that the computer device cluster executes the method according to any one of claims 1 to 11.

26. A retrieval system, characterized in that: The retrieval system includes a client and a cloud device; The cloud device is used to execute the method as described in any one of claims 1 to 11 above; The client is used to execute the method as described in any one of claims 12 to 17 above.

Images