Image generation method and device, electronic device and storage medium

Abstract

This disclosure provides an image generation method, apparatus, electronic device, and storage medium, relating to the field of artificial intelligence technology, particularly natural language processing, deep learning, and large language models. The implementation scheme includes: acquiring current dialogue data, wherein the current dialogue data includes user input data from the current round of dialogue and historical dialogue data from previous rounds of dialogue; determining the user's request type in the current round of dialogue based on the current dialogue data; determining an action sequence for fulfilling the image processing request in response to the request type being an image processing request, wherein the action sequence includes at least one image processing action; executing the action sequence to generate a target image; and generating response data corresponding to the user input data based on the target image. This disclosure enables conversational image generation.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as natural language processing, deep learning, and large language models, and specifically to an image generation method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. AI hardware technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing; AI software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, as well as machine learning / deep learning, big data processing technology, and knowledge graph technology.

[0003] Large Language Model (LLM) is a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of natural language text. Large language models can handle a variety of natural language tasks, such as dialogue, question-answering, text classification, text generation, etc., and are an important path to artificial intelligence. Some large language models also have multimodal data processing capabilities, such as the ability to process multimodal data such as text, images, and videos.

[0004] The methods described in this section are not necessarily methods that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any method described in this section is considered to be prior art simply because it is included in this section. Similarly, unless otherwise indicated, the issues mentioned in this section should not be considered to have been recognized in any prior art. Summary of the invention

[0005] The present disclosure provides an image generating method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, there is provided an image generation method, comprising: acquiring current conversation data, wherein the current conversation data includes user input data of a current round of conversation and historical conversation data of historical rounds of conversation; determining the type of demand of the user in the current round of conversation based on the current conversation data; in response to the demand type being an image processing demand, determining an action sequence for realizing the image processing demand, wherein the action sequence includes at least one image processing action; executing the action sequence to generate a target image; and generating response data corresponding to the user input data based on the target image.

[0007] According to one aspect of the present disclosure, there is provided an image generating device, comprising: an acquisition module, configured to acquire current dialogue data, wherein the current dialogue data includes user input data of a current round of dialogue and historical dialogue data of historical rounds of dialogue; a first determination module, configured to determine the type of demand of the user in the current round of dialogue based on the current dialogue data; a second determination module, configured to determine an action sequence for realizing the image processing demand in response to the demand type being an image processing demand, wherein the action sequence includes at least one image processing action; an execution module, configured to execute the action sequence to generate a target image; and a generation module, configured to generate response data corresponding to the user input data based on the target image.

[0008] According to one aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the above method.

[0009] According to one aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the above method.

[0010] According to one aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which implement the above method when executed by a processor.

[0011] According to one or more embodiments of the present disclosure, conversational image generation can be achieved, thereby improving the efficiency and convenience of image generation.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings exemplarily illustrate the embodiments and constitute a part of the specification, and together with the text description of the specification, are used to explain the exemplary implementation of the embodiments. The embodiments shown are for illustrative purposes only and do not limit the scope of the claims. In all drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0014] Figure 1 A schematic diagram showing an exemplary system in which various methods described herein may be implemented according to an embodiment of the present disclosure;

[0015] Figure 2 A schematic diagram of a conversational image processing system according to an embodiment of the present disclosure is shown;

[0016] Figure 3 A flowchart of an image generating method according to an embodiment of the present disclosure is shown;

[0017] Figure 4 A schematic diagram showing an image generation process according to an embodiment of the present disclosure is shown;

[0018] Figure 5 A schematic diagram showing an image generation effect according to an embodiment of the present disclosure;

[0019] Figure 6 A structural block diagram of a training device for a text generation model according to an embodiment of the present disclosure is shown; and

[0020] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, temporal relationship, or importance relationship of these elements, and such terms are only used to distinguish one element from another element. In some examples, the first element and the second element may refer to the same instance of the element, and in some cases, based on the description of the context, they may also refer to different instances.

[0023] The terms used in the description of various examples in this disclosure are only for the purpose of describing specific examples and are not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the element can be one or more. In addition, the term "and / or" used in this disclosure covers any one of the listed items and all possible combinations. "Multiple" refers to two or more.

[0024] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0025] Image processing tasks can be divided into image generation tasks and image editing tasks.

[0026] For image generation tasks, the AIGC (AI Generated Content) technology field has shown great potential in image generation, which has ignited everyone's interest in a short period of time. At present, AI algorithms can complete image creation within minutes, and the production methods of industries related to image creation are being subverted.

[0027] However, the current mainstream image generation solution is a technology that uses a text-based model to generate images based on input text, namely the "text-based" technology, in which the text input by humans is called a "prompt". Text-based models include text-based models based on Generative Adversarial Network (GAN), text-based models based on Transformer, and text-based models based on diffusion generation (Diffusion). Text-based graph technology requires a clear and finely optimized prompt to obtain relatively ideal results. Therefore, this mainstream technology has a high threshold for abstract use, resulting in low efficiency and high cost of image processing. At the same time, it is often difficult to meet user needs because it cannot support multiple rounds of interaction.

[0028] Image editing tasks are usually accomplished using traditional image editing tools (such as PhotoShop, CorelDRAW, and other software). Users can select and use corresponding functions in image editing tools to independently complete actions such as adding content, deleting content, extracting content, and adjusting the size and color of images. Users need to undergo special training to use image editing tools, which results in low efficiency and high cost of image editing, making it difficult to meet user needs.

[0029] As can be seen from the above, the image processing solutions in the related art are not universal, have a high threshold for use, and do not support or facilitate multiple rounds of interaction with users, resulting in low efficiency and high cost of image processing, and difficulty in meeting user needs.

[0030] In view of the above problems, the embodiments of the present disclosure provide a conversational image generation method, in which the user can complete image processing by interacting with the image processing system in multiple rounds in a conversational manner. The method is easy to use and does not require additional learning costs, which greatly improves the efficiency and convenience of image processing. The embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

[0031] Figure 1 FIG. 1 is a schematic diagram of an exemplary system 100 in which various methods and apparatuses described herein may be implemented according to an embodiment of the present disclosure. Figure 1 , the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0032] In an embodiment of the present disclosure, the client devices 101 , 102 , 103 , 104 , 105 , and 106 and the server 120 may run one or more services or software applications that enable execution of the image generation method.

[0033] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtualized environments and virtualized environments. In some embodiments, these services may be provided as web-based services or cloud services, such as provided to users of client devices 101, 102, 103, 104, 105, and / or 106 under a software as a service (SaaS) model.

[0034] exist Figure 1 In the configuration shown, the server 120 may include one or more components that implement the functions performed by the server 120. These components may include software components, hardware components, or a combination thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 may in turn utilize one or more client applications to interact with the server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from the system 100. Therefore, Figure 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

[0035] Client devices 101, 102, 103, 104, 105, and / or 106 may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Figure 1 Only six client devices are depicted, but one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0036] Client devices 101, 102, 103, 104, 105 and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, vehicle-mounted devices, game systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems; or include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Game systems may include various handheld game devices, Internet-enabled game devices, etc. Client devices are capable of executing various different applications, such as various Internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and may use various communication protocols.

[0037] The network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP / IP, SNA, IPX, etc. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0038] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain a server's virtual storage device). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0039] The computing units in the server 120 may run one or more operating systems including any of the above operating systems and any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and / or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0040] In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0041] In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in a cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0042] The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. The databases 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The databases 130 may be of different types. In some embodiments, the databases used by the server 120 may be, for example, relational databases. One or more of these databases may store, update, and retrieve data to and from the databases in response to commands.

[0043] In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the applications may be different types of databases, such as a key-value store, an object store, or a conventional store backed by a file system.

[0044] Figure 1 The system 100 may be configured and operated in various ways to enable the application of various methods and apparatuses described in the present disclosure.

[0045] Figure 2 A schematic diagram of conversational image processing according to an embodiment of the present disclosure is shown.

[0046] like Figure 2 As shown, user 210 can use a client device (eg Figure 1 The client devices 101-106 shown in FIG. 1 and FIG. 2 communicate with the image processing system 220. The image processing system 220 may be, for example, a server (eg, Figure 1 An application in the server 120) shown in FIG.

[0047] In each round of dialogue, the user 210 sends the input data 230 of the current round to the image processing system 220. The image processing system 220 responds to the input data 230, determines the image processing requirements of the user, and performs corresponding image processing actions to obtain a target image that meets the user's requirements. Furthermore, the image processing system 220 can generate textual interpretation data of the target image, and feed the target image and its interpretation data as response data 240 to the user.

[0048] The user 210 can have multiple rounds of dialogue with the image processing system 220. Accordingly, the image processing system 220 can gradually understand the user's needs during the multiple rounds of dialogue and meet the user's needs through its own image generation and image editing capabilities.

[0049] In the embodiments of the present disclosure, the user 210 expresses his image processing requirements through natural language (specifically, it can be expressed through voice input or text input), without any technical background or learning specific technical knowledge, such as learning the construction and optimization methods of Wenshengtu prompt words. It can be understood that, if necessary, in addition to natural language, various images can also be input as a way to assist in expressing requirements. For example, a user can enter or specify an image in the dialogue interface and express the image processing requirements in text. The following are some examples of demand texts that match images:

[0050] 1. Please help me create in the style of this picture;

[0051] 2. Please refer to the character movements in this picture for creation;

[0052] 3. For the pictures generated in the previous round, the picture is adjusted to be more lively.

[0053] In addition, the user can evaluate the image processing results fed back by the system and further express optimization requirements. In this way, on the one hand, the user 210 can gradually iterate the image processing results to meet his own needs; on the other hand, the image processing system 220 can also record the user's evaluation content as a basis for self-optimization, for example, as a basis for optimizing the large language model, the image processing model called by each image processing action, etc.

[0054] Users can express various image processing requirements, and accordingly, the image processing system 220 has various image processing capabilities. Figure 2 As shown, image processing can be divided into two categories: image generation and image editing.

[0055] Image generation refers to the image processing system 220 creating images according to user requirements, which mainly includes three methods:

[0056] 1. Image generation: Generate new images using natural language prompts as input. This is usually the most common image generation requirement.

[0057] 2. Image-to-image: Generate new images using images as input, such as extracting line drawings.

[0058] 3. Text-image hybrid generation: Generate new images using natural language and images (possibly multiple) as input at the same time. For example, draw a picture based on a text prompt according to the style of reference image A and the composition of reference image B.

[0059] Image editing refers to making specified changes to an existing image (usually an image generated in a historical round of dialogue) to obtain a new image. It can be understood that since a new image is obtained through image editing, image editing is essentially also image generation. Image editing operations include but are not limited to the following five:

[0060] 1. New content: Add new elements at the specified position of the image. In the case of natural language interaction, the position of the new content can be automatically determined by the system (for example: if a moon is added to the image, the position of the moon can be automatically selected).

[0061] 2. Delete content: Delete specified content from the image, which can be specified by position or content.

[0062] 3. Modify content: modify the color, shape, style, etc. of the specified content in the image.

[0063] 4. Extract content: Extract content elements from the image, such as background, foreground, etc.

[0064] 5. Full image adjustment: Do not specify specific content or areas, but make adjustments to the image as a whole, such as style conversion, resolution improvement, aspect ratio modification, etc.

[0065] In the process of image editing, the interactive methods include not only natural language (for example, deleting the sun in the picture), but also interactive operations on the image (for example, using screen sliding gestures to cover the area where new content needs to be added).

[0066] It should be noted that, unlike traditional image editing tools, each image editing capability in the image processing system 220 essentially includes the creation of the entire image or a part of the image. For example, the traditional "cutout" can extract the specified content from the image, but the "extract content" of the present system can also complete the background part through image creation based on the separation of the extracted content.

[0067] Figure 3 1 shows a flow chart of an image generation method 300 according to an embodiment of the present disclosure. The execution subject of each step of the method 300 is usually a server, for example Figure 1 More specifically, the method 300 may be a server based on an image processing system (eg, Figure 2 In some embodiments, the execution subject of method 300 may also be a client device, such as Figure 1 Client devices 101-106 are shown in FIG.

[0068] like Figure 3 As shown, the method 300 includes steps S310 - S350 .

[0069] In step S310, current dialogue data is obtained, wherein the current dialogue data includes user input data of the current dialogue round and historical dialogue data of the previous dialogue rounds.

[0070] In step S320, based on the current dialogue data, the type of user's needs in the current round of dialogue is determined.

[0071] In step S330, in response to the requirement type being an image processing requirement, an action sequence for implementing the image processing requirement is determined, wherein the action sequence includes at least one image processing action.

[0072] In step S340 , an action sequence is executed to generate a target image.

[0073] In step S350, response data corresponding to the user input data is generated based on the target image.

[0074] According to an embodiment of the present disclosure, a conversational image generation method is provided, and a user can perform multiple rounds of interaction with an image processing system in a conversational manner.

[0075] In each round of dialogue, based on the current user input data and context information (i.e., historical dialogue data), the user's image processing requirements are automatically identified, the corresponding action sequence is executed to obtain the target image, and the target image is fed back to the user as the response data of the current round of dialogue.

[0076] Users can complete image processing by having multiple rounds of natural language conversations with the system without the need for special learning, which greatly reduces the usage threshold and improves the efficiency and convenience of image processing.

[0077] The various steps of method 300 are described in detail below.

[0078] Regarding step S310, it can be understood that each historical round of dialogue includes historical input data of the user and historical response data fed back by the image processing system in response to the historical input data.

[0079] According to some embodiments, the current conversation data, user input data and historical conversation data in step S310 may include at least one of text, voice or image. It is understood that when the current conversation data, user input data and historical conversation data are voice, the voice may be converted into text using voice recognition technology, thereby simplifying subsequent data processing steps and improving computing efficiency.

[0080] According to some embodiments, step S320 may include steps S321 and S322.

[0081] In step S321, first input data for inputting a first language model is determined based on the current dialog data.

[0082] In step S322, the first input data is input into the first language model to obtain the requirement type output by the first language model.

[0083] The first language model is a large language model, which usually includes an N-layer Transformer network with an encoder and a decoder. The large language model is obtained by pre-training with a large amount of natural language data. Pre-training enables the large language model to have a large amount of prior knowledge and common sense, thereby improving its performance on various tasks.

[0084] According to the above embodiment, using a pre-trained large language model to identify user needs can improve the accuracy and flexibility of demand identification.

[0085] According to some embodiments, in step S321, the current conversation data may be directly used as the first input data.

[0086] According to some other embodiments, in step S321, a preset first template may be used to generate the first input data. Specifically, step S321 may include steps S3211 and S3212.

[0087] In step S3211, a preset first template is obtained. The first template includes first guidance information for guiding the first language model to identify the requirement type and a first slot to be filled.

[0088] In step S3212, the current conversation data is filled into the first slot to obtain the first input data.

[0089] According to the above embodiment, the accuracy of demand identification can be improved by using the preset first template to guide the large language model to identify user needs.

[0090] According to some embodiments, the first template may include a first slot for filling the current conversation data and a first slot for filling the candidate demand type. This can guide the large language model to select a suitable demand type from multiple candidate demand types based on the current conversation data, thereby avoiding the large language model generating unknown demand types and causing adverse effects on subsequent image processing steps.

[0091] For example, the first template may be, for example, ““__”. Based on the above conversation content, please select an appropriate requirement type in “__”. The underline in the first template indicates the first slot to be filled, wherein the first first slot is used to fill the current conversation data, and the second second slot is used to fill multiple candidate requirement types. The other characters in the first template except the underline are the first guide information.

[0092] According to some embodiments, after the first input data is input into the first language model, the first language model can not only identify and output the demand type, but also extract and output the target data for achieving the demand type from the current conversation data. It can be understood that for different demand types, the data content that needs to be paid attention to may be different. For example, the current conversation data may include the user's chat data "I'm so bored, come and chat with me", which is redundant information that does not need to be paid attention to for image processing requirements. According to the above embodiment, by extracting the target data, irrelevant information in the current conversation data can be filtered out, thereby improving the efficiency and accuracy of image processing.

[0093] Table 1 below shows some examples of requirement types and their corresponding target data.

[0094] Table 1

[0095]

[0096]

[0097] According to some embodiments, step S320 may include step S323.

[0098] In step S323, the current dialog data is input into the classification model to obtain the demand type output by the classification model.

[0099] The classification model may be, for example, a text classification model, an image classification model, etc. The classification model may map the text or image data in the current conversation data to one of the candidate multiple demand types. The classification model may be, for example, a neural network model. Compared with a large language model, the classification model has fewer parameters and is more lightweight. The classification model can be used to quickly determine the demand type.

[0100] If the requirement type determined in step S320 is an image processing requirement, then a sequence of actions to be performed to implement the image processing requirement is further determined.

[0101] In the embodiments of the present disclosure, image processing requirements refer to requirements related to image processing. For example, among the multiple types of requirements shown in Table 1, image generation requirements, image editing requirements, image optimization requirements, and image creation evaluation requirements are all image processing requirements.

[0102] The action sequence includes at least one image processing action, and at least one image processing action in the action sequence needs to be executed in a certain order. Two examples of action sequences are given below:

[0103] 1. First, optimize the user's prompt, then generate 4 image results, and finally select the best one to feed back to the user.

[0104] 2. First, identify the area specified by the user's editing needs, and then complete the partial screen redrawing.

[0105] According to some embodiments, step S330 may include steps S331 and S332.

[0106] In step S331 , based on image processing requirements, second input data for inputting into a second language model is determined.

[0107] In step S332, the second input data is input into the second language model to obtain an action sequence output by the second language model.

[0108] The second language model is a pre-trained large language model. In some embodiments, the second language model and the first language model above are the same large language model. According to the above embodiment, using the pre-trained large language model to generate the action sequence can improve the accuracy and flexibility of the action sequence planning.

[0109] According to some embodiments, step S330 may include steps S333 - S335 .

[0110] In step S333, a preset second template is obtained, wherein the second template includes second guiding information for guiding the second language model to generate the action sequence and a second slot to be filled. The second template may be, for example, "How to implement __?", wherein the underscore indicates the second slot for filling the image processing requirement, and the other characters are the second guiding information.

[0111] In step S335, the image processing requirement is filled into the second slot to obtain second input data for inputting the second language model.

[0112] In step S336, the second input data is input into the second language model to obtain the action sequence output by the second language model.

[0113] According to the above embodiment, the action sequence is automatically planned using a large language model, which has high accuracy and flexibility.

[0114] According to some embodiments, a correspondence between multiple image processing requirements and multiple action sequences may be preset. Accordingly, in step S330, an action sequence for achieving the image processing requirements may be determined based on the preset correspondence between multiple image processing requirements and multiple action sequences. Thus, the action sequence can be quickly obtained without performing additional calculations.

[0115] It should be noted that, in the above preset corresponding relationship, one image processing requirement may correspond to multiple action sequences. Accordingly, in step S330, any one of the multiple action sequences corresponding to the image processing requirement may be selected and executed in step S340.

[0116] The action sequence includes at least one image processing action. Table 2 shows some examples of image processing actions.

[0117] Table 2

[0118]

[0119]

[0120]

[0121] According to some embodiments, step S340 may include: extracting target data for realizing the image processing requirement from the current dialog data; for any image processing action in the action sequence: determining an input parameter value of the image processing action based on the target data; and executing the image processing action based on the input parameter value to obtain a result image of the image processing action. The input parameters of each image processing action may refer to Table 2 above, for example.

[0122] According to the above embodiment, key target data is extracted from the current conversation data, and the image processing action is performed using the target data, which can filter out irrelevant information and improve the efficiency and accuracy of image processing.

[0123] According to some embodiments, the target data may be extracted from the first language model. That is, in step S320, by inputting the first input data into the first language model, the image processing requirement output by the first language model and the target data for achieving the image processing requirement may be obtained.

[0124] According to some embodiments, the image processing action may be performed by calling the corresponding image processing model. For example, the image generation action may be completed by calling the cross-modal image generation model, and the image editing action may be completed by calling the cross-modal image editing model to obtain a result image.

[0125] The cross-modal image generation model takes text and images as input and images as output, and the input is used as a condition for generating images. The embodiments of the present disclosure do not have strict requirements on the specific implementation of the cross-modal image generation model. Under current technical conditions, the model with the best effect is the cross-modal image generation model based on the diffusion generation model. With the development of technology, this model can also be replaced by a better model. It should be noted that this model is not a common "text image" model. The input end of the model can be text, images and their mixtures. The image may be a reference image, a pad image or a feature image, and the input form is flexible.

[0126] The cross-modal image editing model takes text, images, and editing options as input, and makes specified editing changes to the input image. The embodiments of the present disclosure do not have strict requirements on the specific implementation requirements of the cross-modal image editing model, and even different editing options can correspond to different editing models. However, the capabilities provided by the image editing model should be able to cover the optional editing actions in the action flow planning.

[0127] In both models, text input needs to be processed, that is, the text input is understood as a text vector and involved in subsequent calculations. This requires the understanding ability of a large language model.

[0128] In step S350 , response data corresponding to the user input data is generated based on the target image.

[0129] According to some embodiments, the target image may be directly fed back to the user as response data.

[0130] According to other embodiments, the response data includes the target image and the interpretation data of the target image. The interpretation data includes, for example, the description of the target image, the process and logic of the system generating the target image, etc. Compared with only using the target image as the response data, feeding back the target image and its interpretation data to the user together can more effectively explain to the user the content of the target image and the process and logic of the system generating the target image, thereby facilitating the user to perform targeted optimization of the target image later.

[0131] According to some embodiments, step S350 may include steps S351 and S352.

[0132] In step S351, a target image and a preset third template are input into a third language model to obtain interpretation data output by the third language model for interpreting the target image, wherein the third template is used to guide the third language model to generate the interpretation data.

[0133] In step S352, the target image and the interpretation data are used as the response data.

[0134] The third language model is a large language model with the ability to process cross-modal data of images and texts. That is, the input and data of the large language model can be images, texts, or a mixture of the two. According to the above embodiment, using a large language model with the ability to process cross-modal data of images and texts to generate response data can improve the generation efficiency and consistency of response data.

[0135] According to some embodiments, the third language model may be the same model as the first language model or the second language model mentioned above.

[0136] According to some embodiments, the third template may be, for example, “Please describe the content and artistry of this picture, and ask the user about his / her feelings about this picture and the optimization direction.”

[0137] According to some embodiments, in step S351, the action sequence, the target image and the third template may be input into the third language model together, so that the third language model generates explanatory data in combination with the image processing results and the action flow process, and can more effectively explain the process and logic of the system output to the user, thereby facilitating the user to perform targeted optimization of the target image in the next round of dialogue.

[0138] According to some embodiments, step S350 may include steps S353 - S355 .

[0139] In step S353, the target image is input into the image-to-text model to obtain a description text of the target image output by the image-to-text model.

[0140] In step S354, the description text is input into a fourth language model to obtain interpretation data output by the fourth language model for interpreting the target image. The fourth language model is a pre-trained large language model.

[0141] In step S355, the target image and the interpretation data are used as the response data.

[0142] According to the above embodiment, the description text of the target image is first generated by the image-to-text model, and then the response data is generated by the large language model. This embodiment can be applied to the large language model that only has the ability to process single-modal (text) data.

[0143] According to some embodiments, the image-to-text model may be implemented as a Transformer structure including an image encoder (Image Encoder) and a text decoder (Text Decoder).

[0144] According to some embodiments, the fourth language model may be the same model as the first language model, the second language model, or the third language model described above.

[0145] Figure 4 FIG. 1 is a schematic diagram showing an image generation process according to an embodiment of the present disclosure. Figure 4 As shown, the image generation process includes steps S461-S464.

[0146] In step S461, in each round of dialogue, the interaction context 410 (ie, current dialogue data) between the current user and the image processing system is obtained, and the large language model 420 is used to understand the user needs expressed in the interaction context 410, thereby achieving multi-round demand understanding.

[0147] In step S462 , the large language model 420 plans an action flow (ie, an action sequence) based on the user requirements understood in step S461 .

[0148] In step S463 , the action flow is executed by calling the cross-modal image generation model 430 and / or the cross-modal image editing model 440 to obtain a target image.

[0149] In step S464 , the large language model 420 generates description data corresponding to the target image, and feeds back the target image and its description data to the user 450 .

[0150] Figure 5 FIG. 1 is a schematic diagram showing an image generation effect according to an embodiment of the present disclosure. Figure 5 In the illustrated embodiment, the user has multiple rounds of dialogue with the image processing system.

[0151] In the first round of dialogue, the user inputs content 500 in the dialogue interface, "Just generate a picture for me." In response to the content 500, the image processing system generates a target image 512 and its interpretation data using the method 300 of the embodiment of the present disclosure, "I generated a picture, you can find it at images / aaa.png." The combination of the target image 512 and its interpretation data is used as a response 510 to the user input content 500.

[0152] In the second round of dialogue, the user inputs content 520 in the dialogue interface, "Tell me what you know about Star Wars". In response to the content 520, the image processing system generates a target image 532 and its interpretation data using the method 300 of the disclosed embodiment, "I generated a picture about Star Wars, which you can find at images / bbb.png. As for your question, Star Wars is a science fiction movie that tells the battles and adventures of various forces in the Milky Way." The combination of the target image 532 and its interpretation data is used as a response 530 to the user input content 520.

[0153] In the third round of dialogue, the user enters content 540 in the dialogue interface, "Change the background of the Star Wars picture you just generated to a starry sky." In response to the content 540, the image processing system generates a target image 552 and its interpretation data using the method 300 of the embodiment of the present disclosure, "I have used an image editing tool to change the background of the Star Wars picture to a starry sky. The path of the new image is image / ccc.png." The combination of the target image 552 and its interpretation data is used as a response 550 to the user input content 540.

[0154] In the fourth round of dialogue, the user enters content 560 in the dialogue interface, "Make the colors of the whole picture more vivid." In response to content 560, the image processing system generates a target image 572 and its interpretation data, "The colors of the picture have been made more vivid using the image editing tool, and the path of the new image is image / ddd.png.", using the method 300 of the embodiment of the present disclosure. The combination of the target image 572 and its interpretation data is used as a response 570 to the user input content 560.

[0155] In the fifth round of dialogue, the user enters content 580 in the dialogue interface: “No, I mean to make the color of the Star Wars picture more vivid.” In response to content 580, the image processing system generates a target image 592 and its interpretation data using the method 300 of the embodiment of the present disclosure: “Okay, the color of the Star Wars picture has been made more vivid using the image editing tool. The path of the new image is image / eee.png.” The combination of the target image 592 and its interpretation data is used as a response 590 to the user input content 580.

[0156] According to the embodiments of the present disclosure, the image processing system can conduct unlimited rounds of interactions with users, continuously guide users to express complete and accurate real needs during the interactions, and gradually approach the ideal result that meets user needs in multiple steps. It can complete both image generation tasks and image editing tasks, significantly improving the versatility of AI image creation applications.

[0157] The image processing system of the disclosed embodiment leverages the capabilities of a large language model. Users do not need to conduct specialized learning and can communicate with the system using natural language, which greatly reduces the threshold for using AI image creation.

[0158] In each step or module using a large language model, the effect of using a large language model can be further improved through prompt optimization, In-context learning, etc. For example, you can first call the large language model without using context examples to evaluate the processing effect of the large language model on various types of tasks. Construct context examples for task types with poor results to improve the processing effect of the large language model on these types of tasks.

[0159] According to an embodiment of the present disclosure, an image generating device is also provided. Figure 6 FIG. 6 shows a structural block diagram of an image generating device 600 according to an embodiment of the present disclosure. Figure 6 As shown, the apparatus 600 includes an acquisition module 610 , a first determination module 620 , a second determination module 630 , an execution module 640 and a generation module 650 .

[0160] The acquisition module 610 is configured to acquire current dialogue data, wherein the current dialogue data includes user input data of a current dialogue round and historical dialogue data of historical dialogue rounds.

[0161] The first determination module 620 is configured to determine the demand type of the user in the current round of dialogue based on the current dialogue data.

[0162] The second determination module 630 is configured to determine, in response to the requirement type being an image processing requirement, an action sequence for implementing the image processing requirement, wherein the action sequence includes at least one image processing action.

[0163] The execution module 640 is configured to execute the action sequence to generate a target image.

[0164] The generating module 650 is configured to generate response data corresponding to the user input data based on the target image.

[0165] According to an embodiment of the present disclosure, a conversational image generation method is provided, and a user can perform multiple rounds of interaction with an image processing system in a conversational manner.

[0166] In each round of dialogue, based on the current user input data and context information (i.e., historical dialogue data), the user's image processing requirements are automatically identified, the corresponding action sequence is executed to obtain the target image, and the target image is fed back to the user as the response data of the current round of dialogue.

[0167] Users can complete image processing by having multiple rounds of natural language conversations with the system without the need for special learning, which greatly reduces the usage threshold and improves the efficiency and convenience of image processing.

[0168] According to some embodiments, the first determination module includes: a first determination unit, configured to determine first input data for inputting into a first language model based on the current conversation data; and a first input unit, configured to input the first input data into the first language model to obtain the demand type output by the first language model.

[0169] According to some embodiments, the first determination unit includes: an acquisition subunit, configured to acquire a preset first template, wherein the first template includes first guidance information for guiding the first language model to identify the requirement type and a first slot to be filled; and a filling subunit, configured to fill the current conversation data into the first slot to obtain the first input data.

[0170] According to some embodiments, the first determination module includes: a second input unit configured to input the current conversation data into a classification model to obtain the demand type output by the classification model.

[0171] According to some embodiments, the second determination module includes: an acquisition unit, configured to acquire a preset second template, wherein the second template includes second guidance information for guiding the second language model to generate the action sequence and a second slot to be filled; a filling unit, configured to fill the image processing requirements into the second slot to obtain second input data for inputting the second language model; and a third input unit, inputting the second input data into the second language model to obtain the action sequence output by the second language model.

[0172] According to some embodiments, the second determination module includes: a second determination unit configured to determine an action sequence for realizing the image processing requirement based on a correspondence relationship between a plurality of preset image processing requirements and a plurality of action sequences.

[0173] According to some embodiments, the execution module includes: an extraction unit, configured to extract target data for realizing the image processing requirement from the current conversation data; a third determination unit, configured to determine, for any image processing action in the action sequence, an input parameter value of the image processing action based on the target data; and an execution unit, configured to execute the image processing action based on the input parameter value to obtain a result image of the image processing action.

[0174] According to some embodiments, the generation module includes: a fourth input unit, configured to input the target image and a preset third template into a third language model to obtain interpretation data output by the third language model for interpreting the target image, wherein the third template is used to guide the third language model to generate the interpretation data; and a fourth determination unit, configured to use the target image and the interpretation data as the response data.

[0175] According to some embodiments, the generation module includes: a fifth input unit, configured to input the target image into a graphic-text model to obtain a description text of the target image output by the graphic-text model; a sixth input unit, configured to input the description text into a fourth language model to obtain interpretation data output by the fourth language model for interpreting the target image; and a fifth determination unit, configured to use the target image and the interpretation data as the response data.

[0176] It should be understood that Figure 6 The various modules and units of the apparatus 600 shown in FIG. 6 can be compared with those of the reference Figure 3 The steps in the method 300 described above correspond to each other. Therefore, the operations, features and advantages described above for the method 300 are also applicable to the device 600 and the modules and units included therein. For the sake of brevity, some operations, features and advantages are not repeated here.

[0177] Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be separated into multiple modules, and / or at least some functionality of multiple modules may be combined into a single module.

[0178] It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. Figure 6The various units described can be implemented in hardware or in hardware in combination with software and / or firmware. For example, these units can be implemented as computer program codes / instructions, which are configured to be executed in one or more processors and stored in a computer-readable storage medium. Alternatively, these units can be implemented as hardware logic / circuits. For example, in some embodiments, one or more of modules 610-650 can be implemented together in a system on chip (System on Chip, SoC). SoC can include an integrated circuit chip (which includes a processor (e.g., a central processing unit (CPU), a microcontroller, a microprocessor, a digital signal processor (DSP), etc.), a memory, one or more communication interfaces, and / or one or more components in other circuits), and can optionally execute the received program code and / or include embedded firmware to perform functions.

[0179] According to an embodiment of the present disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so that the at least one processor can execute the image generation method of the embodiment of the present disclosure.

[0180] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided. The computer instructions are used to enable a computer to execute the image generating method of the embodiment of the present disclosure.

[0181] According to an embodiment of the present disclosure, a computer program product is further provided, including computer program instructions, which implement the image generating method of the embodiment of the present disclosure when executed by a processor.

[0182] refer to Figure 7 , a block diagram of an electronic device 700 that can be used as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0183] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0184] Multiple components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device that can input information to the electronic device 700. The input unit 706 can receive input digital or character information and generate key signal input related to user settings and / or function control of the electronic device, and can include but is not limited to a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and / or a remote controller. The output unit 707 can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 can include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0185] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as method 300 or method 300. For example, in some embodiments, method 300 and method 300 may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of method 300 and method 300 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the method 300 or the method 300 in any other suitable manner (eg, by means of firmware).

[0186] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0187] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0188] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0189] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0190] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0191] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0192] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be performed in parallel, sequentially or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here.

[0193] Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-mentioned methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but is only limited by the claims after authorization and their equivalent scope. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that with the evolution of technology, many of the elements described herein may be replaced by equivalent elements that appear after the present disclosure.

Claims

1. A method for generating an image, comprising: Acquire current dialogue data, wherein the current dialogue data includes user input data of the current dialogue round and historical dialogue data of the historical dialogue rounds; Determining the type of demand of the user in the current round of conversation based on the current conversation data; In response to the requirement type being an image processing requirement, determining an action sequence for implementing the image processing requirement, wherein determining the action sequence for implementing the image processing requirement includes: Acquire a preset second template, wherein the second template includes second guide information for guiding the second language model to generate the action sequence and a second slot to be filled; Filling the image processing requirement into the second slot to obtain second input data for inputting the second language model; and Inputting the second input data into the second language model to obtain the action sequence output by the second language model, wherein the action sequence includes a plurality of image processing actions in a set order, and the image processing actions include: optimizing the required text for generating an image, retrieving a reference image, generating an image from text, generating an image from an image, generating an image from a mixture of text and image, adding an image element, modifying an image element, deleting an image element, superimposing an image, optimizing the meta information of an image, optimizing the overall effect of an image, selecting the best image from multiple images, and collecting image evaluations; According to the set sequence, the plurality of image processing actions are executed by calling corresponding image processing models to generate a target image; and Based on the target image, response data corresponding to the user input data is generated.

2. The method according to claim 1, wherein: The determining, based on the current conversation data, the type of demand of the user in the current round of conversation includes: Based on the current dialog data, determining first input data for inputting into a first language model; and The first input data is input into the first language model to obtain the requirement type output by the first language model.

3. The method according to claim 2, wherein: The determining, based on the current dialog data, first input data for inputting into the first language model comprises: Acquire a preset first template, wherein the first template includes first guidance information for guiding the first language model to identify the requirement type and a first slot to be filled; and The current conversation data is filled into the first slot to obtain the first input data.

4. The method according to claim 1, wherein: The determining, based on the current conversation data, the type of demand of the user in the current round of conversation includes: The current conversation data is input into a classification model to obtain the demand type output by the classification model.

5. The method according to any one of claims 1 to 4, wherein: The determining of an action sequence for achieving the image processing requirement comprises: Based on the correspondence between a plurality of preset image processing requirements and a plurality of action sequences, an action sequence for realizing the image processing requirements is determined.

6. The method according to claim 1, wherein: The step of performing the plurality of image processing actions by calling corresponding image processing models in the set order to generate a target image includes: Extracting target data for achieving the image processing requirement from the current conversation data; For any image processing action in the action sequence: Based on the target data, determining input parameter values ​​of the image processing action; and Based on the input parameter value, the image processing action is performed to obtain a result image of the image processing action.

7. The method according to claim 1, wherein: The generating, based on the target image, response data corresponding to the user input data comprises: Inputting the target image and a preset third template into a third language model to obtain interpretation data output by the third language model for interpreting the target image, wherein the third template is used to guide the third language model to generate the interpretation data; and The target image and the interpretation data are used as the response data.

8. The method according to claim 1, wherein: The generating, based on the target image, response data corresponding to the user input data comprises: Inputting the target image into a picture-to-text model to obtain a description text of the target image output by the picture-to-text model; Inputting the description text into a fourth language model to obtain interpretation data output by the fourth language model for interpreting the target image; and The target image and the interpretation data are used as the response data.

9. An image generating device, comprising: An acquisition module is configured to acquire current dialogue data, wherein the current dialogue data includes user input data of a current dialogue round and historical dialogue data of historical dialogue rounds; A first determination module is configured to determine the type of demand of the user in the current round of dialogue based on the current dialogue data; The second determining module is configured to determine, in response to the requirement type being an image processing requirement, an action sequence for implementing the image processing requirement, wherein the second determining module includes: an acquiring unit, configured to acquire a preset second template, wherein the second template includes second guiding information for guiding the second language model to generate the action sequence and a second slot to be filled; a filling unit configured to fill the image processing requirement into the second slot to obtain second input data for inputting the second language model; and a third input unit, inputting the second input data into the second language model to obtain the action sequence output by the second language model, wherein the action sequence includes a plurality of image processing actions in a set order, and the image processing actions include: optimizing the required text for generating an image, retrieving a reference image, generating an image from text, generating an image from an image, generating an image from a text-image hybrid, adding an image element, modifying an image element, deleting an image element, superimposing an image, optimizing the meta information of an image, optimizing the overall effect of an image, selecting the best image from multiple images, and collecting image evaluations; an execution module configured to execute the plurality of image processing actions in accordance with the set sequence by calling corresponding image processing models to generate a target image; and The generating module is configured to generate response data corresponding to the user input data based on the target image.

10. The device according to claim 9, wherein: The first determining module comprises: A first determining unit is configured to determine first input data for inputting into a first language model based on the current dialogue data; and The first input unit is configured to input the first input data into the first language model to obtain the requirement type output by the first language model.

11. The device according to claim 10, wherein: The first determining unit includes: an acquisition subunit, configured to acquire a preset first template, wherein the first template includes first guidance information for guiding the first language model to identify the requirement type and a first slot to be filled; and The filling subunit is configured to fill the current conversation data into the first slot to obtain the first input data.

12. The device according to claim 9, wherein: The first determining module comprises: The second input unit is configured to input the current dialogue data into the classification model to obtain the demand type output by the classification model.

13. The device according to any one of claims 9 to 12, wherein: The second determining module comprises: The second determining unit is configured to determine an action sequence for realizing the image processing requirement based on a correspondence relationship between a plurality of preset image processing requirements and a plurality of action sequences.

14. The device according to claim 9, wherein: The execution module includes: an extraction unit, configured to extract target data for achieving the image processing requirement from the current conversation data; A third determining unit is configured to determine, for any image processing action in the action sequence, an input parameter value of the image processing action based on the target data; and The execution unit is configured to execute the image processing action based on the input parameter value to obtain a result image of the image processing action.

15. The device according to claim 9, wherein: The generation module comprises: a fourth input unit configured to input the target image and a preset third template into a third language model to obtain interpretation data output by the third language model for interpreting the target image, wherein the third template is used to guide the third language model to generate the interpretation data; and The fourth determining unit is configured to use the target image and the interpretation data as the response data.

16. The device according to claim 9, wherein: The generation module comprises: a fifth input unit, configured to input the target image into the image-to-text model to obtain a description text of the target image output by the image-to-text model; a sixth input unit configured to input the description text into a fourth language model to obtain interpretation data output by the fourth language model for interpreting the target image; and A fifth determination unit is configured to use the target image and the interpretation data as the response data.

17. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-8.

19. A computer program product comprising computer program instructions, wherein: The computer program instructions implement the method of any one of claims 1 to 8 when executed by a processor.

Images