Image editing method, device, electronic equipment and storage medium

Abstract

To provide a multi-round dialog type image editing method and device, an electronic equipment, a computer-readable storage medium and a computer program product for significantly reducing the complexity of user operations, improving image editing efficiency, and enhancing user experience.SOLUTION: The image editing method comprises: obtaining an editing instruction input by a user in a current round of dialogue and historical dialogue information in a historical round of dialogue, the historical dialogue information comprising a historical dialogue text and at least one historical image; determining a to-be-edited source image from the at least one historical image based on the editing instruction and the historical dialogue information; and editing the source image based on the editing instruction to generate a target image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of natural language processing, computer vision, deep learning, etc., and specifically to an image editing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. [Background technology]

[0002] Artificial Intelligence (AI) is a field that studies how computers can imitate some human thought processes and intelligent behaviors (e.g., learning, reasoning, thinking, planning, etc.), and includes both hardware and software technologies. AI hardware technologies generally include sensors, AI-dedicated chips, cloud computing, distributed storage, big data processing, etc., while AI software technologies mainly include computer vision technology, voice recognition technology, natural language processing technology, machine learning / deep learning, big data processing technology, knowledge graph technology, and other major directions.

[0003] A large language model (LLM, also called a large model) is a deep learning model trained using a large amount of text data, which can generate natural language text or understand the meaning of natural language text. Large language models can handle multiple types of natural language tasks, such as dialogue, text classification, text generation, etc., and are one of the important paths to artificial intelligence. Some large language models also have multimodal data processing capabilities, and can process multimodal data, such as text, images, and videos.

[0004] The approaches described in this section are not necessarily approaches that have been previously conceived or adopted. Unless otherwise noted, any approach described in this section should not be considered prior art merely for its inclusion in this section. Similarly, unless otherwise noted, the subject matter referred to in this section should not be considered to be admitted in the prior art. Summary of the Invention

[0005] The present disclosure provides an image editing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0006] According to one aspect of the present disclosure, there is provided an image editing method, the method including: obtaining an editing instruction input by a user in a dialogue of a current round and historical dialogue information in a dialogue of a historical round, the historical dialogue information including a historical dialogue text and at least one historical image; determining a source image to be edited from the at least one historical image based on the editing instruction and the historical dialogue information; and editing the source image based on the editing instruction to generate a target image.

[0007] According to one aspect of the present disclosure, an image editing device is provided, the device including: an acquisition module configured to acquire editing instructions input by a user in a dialogue of a current round and historical dialogue information in a dialogue of a historical round, the historical dialogue information including historical dialogue text and at least one historical image; a determination module configured to determine a source image to be edited from the at least one historical image based on the editing instructions and the historical dialogue information; and an editing module configured to edit the source image based on the editing instructions to generate a target image.

[0008] According to one aspect of the present disclosure, an electronic device is provided, the electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0009] According to one aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the above method.

[0010] According to one aspect of the present disclosure, there is provided a computer program product including computer program instructions which, when executed by a processor, implement the above method.

[0011] According to one or more embodiments of the present disclosure, multi-round interactive image editing can be realized, which significantly reduces the complexity of user operations and improves image editing efficiency and user experience.

[0012] It should be understood that the subject matter described in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the disclosure will be readily apparent from the following specification. [Brief description of the drawings]

[0013] The drawings illustratively illustrate the examples and constitute a part of the specification and, together with the textual description of the specification, serve to explain exemplary embodiments of the examples. The examples shown are for illustrative purposes only and are not intended to limit the scope of the claims. In all the drawings, the same reference numerals refer to similar, but not necessarily identical, elements. [Figure 1]1 is a schematic diagram of an example system capable of implementing the methods described herein, according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of an image editing method according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 is a schematic diagram of the t-th iteration of the text-to-image diffusion model according to an embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of an image editing process according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a schematic diagram of an example of image editing according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of a multi-round interactive image editing effect according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a structural block diagram of an image editing device according to an embodiment of the present disclosure. [Figure 8] FIG. 1 is a structural block diagram of an example electronic device that can be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, exemplary embodiments of the present disclosure will be described in conjunction with the drawings, and in order to facilitate understanding, various details of the embodiments of the present disclosure will be included therein, but they should be regarded as merely exemplary. Therefore, it should be recognized by those skilled in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for clarity and conciseness, the following description omits the description of known functions and structures.

[0015] In this disclosure, unless otherwise specified, the terms "first", "second", and the like, used to describe various elements are not intended to limit the location, timing, or importance of these elements. Such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, or in some cases, may refer to different instances based on the context.

[0016] The terms used in the description of various examples of the present disclosure are intended only to describe specific examples and are not intended to be limiting. Unless otherwise clearly indicated by the context, an element may be one or more, unless the number of elements is specifically limited. It should be noted that the term "and / or" used in the present disclosure covers any one of the listed items and all possible combinations. "Plurality" means more than one.

[0017] In the technical solution disclosed herein, the acquisition, storage and application of relevant user personal information shall all comply with the provisions of relevant laws and regulations and shall not violate public order and good customs.

[0018] Image editing involves making specified changes to an existing image, such as adjusting brightness or contrast, adding, modifying or deleting elements in the image, to obtain a new image.

[0019] In the related art, images are often edited using traditional image editing tools, such as graphic software such as PhotoShop, CorelDRAW, etc. These image editing tools have high usage requirements, require users to undergo special training, and have complicated and cumbersome operation flows, resulting in low image editing efficiency, high costs, and difficulty in meeting user needs.

[0020] With the development of artificial intelligence technology, generative image editing techniques, such as image inpainting model, image outpainting model, etc., have shown great potential in image editing tasks. Compared with traditional image editing tools, generative image editing techniques have effectively improved image editing efficiency, but to achieve the desired editing effect, users still need to perform cumbersome and specialized operation steps. For example, for generative image inpainting models, users need to specify the image area to be edited in the manner of smearing, and then input empirical and meticulously crafted text (prompt), and the usage methods differ according to the editing purpose (such as adding, modifying, deleting, etc.). To achieve the desired editing effect, users need to understand the relevant algorithm principles, parameter settings, and other specialized knowledge. Therefore, generative image editing techniques are still difficult for users to use, have low image editing efficiency, and are difficult to meet users' needs.

[0021] It can be seen from this that the image editing solutions in the related art lack versatility, have high user requirements, have complicated and cumbersome operation flows, and have low image editing efficiency, which makes it difficult to meet users' needs.

[0022] In response to the above problems, the embodiment of the present disclosure provides a multi-round interactive image editing method. By combining historical interaction information, the editing object (i.e., source image) directed by the user's current editing command is accurately understood, and the source image is edited and a target image is generated according to the user's editing command. The embodiment of the present disclosure understands and meets the user's image editing needs through a unified and natural multi-round interaction manner, significantly reducing the complexity of user operation, and improving image editing efficiency and user experience.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0024] 1 illustrates a schematic diagram of an example system 100 in which various methods and apparatus described herein may be implemented, according to embodiments of the present disclosure. With reference to FIG. 1, the system 100 includes one or more client devices 101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

[0025] In an embodiment of the present disclosure, the client devices 101, 102, 103, 104, 105, and 106 and the server 120 may be operable to perform an image editing method or one or more services or software applications for the image editing method.

[0026] In some embodiments, server 120 may further provide other services or software applications, which may include non-virtualized and virtualized environments. In some embodiments, these services may be provided as web-based or cloud services, for example, provided to users of client devices 101, 102, 103, 104, 105, and / or 106 in a Software as a Service (SaaS) model.

[0027] In the configuration shown in FIG. 1, the server 120 may include one or more assemblies that implement the functions performed by the server 120. These assemblies may include software assemblies, hardware assemblies, or a combination thereof that may be executed by one or more processors. Users operating the client devices 101, 102, 103, 104, 105, and / or 106 may utilize the services provided by these assemblies by, in turn, utilizing one or more client applications to interact with the server 120. It should be understood that a variety of different system configurations are possible and may differ from system 100. As such, FIG. 1 is an example of a system for implementing various methods described herein and is not intended to be limiting.

[0028] Client devices 101, 102, 103, 104, 105, and / or 106 may provide an interface to a user of the client device through which the user can interact with the client device. The client device may also output information to the user through the interface. Although only six client devices are shown in FIG. 1, one skilled in the art will appreciate that the present disclosure may support any number of client devices.

[0029] The client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computing devices, such as portable handheld devices, general purpose computers (e.g., personal computers or laptop computers), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, in-vehicle devices, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices, etc. These computing devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-based operating systems, Linux or Linux-based operating systems, etc., and may include various mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android, etc. Portable handheld devices may include cellular phones, smartphones, tablet PCs, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (e.g., smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, etc. The client devices may run a variety of different applications, such as various Internet-related applications, communication applications (eg, email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.

[0030] Network 110 may be any type of network known to those skilled in the art that may use any one of a number of available protocols (including, but not limited to, TCP / IP, SNA, IPX, etc.) to support data communications. By way of example, one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token loop, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, Wi-Fi), and / or any combination of these and / or other networks.

[0031] Server 120 may include one or more general purpose computers, dedicated server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may also include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage devices that may be virtualized to maintain virtual storage devices for the server). In various embodiments, server 120 may execute one or more services or software applications that provide the functionality described below.

[0032] The computing units in server 120 may run one or more operating systems, including any of the operating systems listed above, as well as any commercially available server operating system. Server 120 may further run any one of a variety of additional server applications and / or mid-tier applications, including an HTTP server, an FTP server, a CGI server, a JAVA server, a database server, etc.

[0033] In some embodiments, server 120 may include one or more applications for integrating with analytics data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105, and / or 106. Server 120 may further include one or more applications for displaying data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105, and / or 106.

[0034] In some embodiments, the server 120 may be a server of a distributed system or a server incorporating blockchain. The server 120 may be a cloud server, or a smart cloud computing server or a smart cloud host equipped with artificial intelligence technology. The cloud server is a host product in the cloud computing service system, which solves the deficiencies of traditional physical host and virtual private server (VPS) services, such as high management difficulty and low business scalability.

[0035] System 100 may further include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of databases 130 may be used to store information for audio and video files. Databases 130 may be located in a variety of locations. For example, a database used by server 120 may be local to server 120 or may be separate from server 120 and in communication with server 120 over a network or dedicated connection. Databases 130 may be of various types. In some embodiments, a database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from the database and from the database in response to instructions.

[0036] In some embodiments, one or more of databases 130 may also be used by an application to store data for the application. The databases used by the application may be different types of databases, such as a key-value repository, an object repository, or a general-purpose repository supported by a file system.

[0037] The system 100 of FIG. 1 may be configured and operated in a variety of ways to accommodate the various methods and apparatus described in this disclosure.

[0038] According to some embodiments, the client devices 101-106 may execute the image editing method of the embodiment of the present disclosure to provide an immersive image editing service for a user. Specifically, the user may communicate his / her image creation or image editing needs by operating the client devices 101-106 (e.g., operating an input device such as a mouse, keyboard, or touch screen) to input an image processing command (i.e., a query) for each round of dialogue. When the user communicates the need for image creation, the client devices 101-106 generate a new target image for the user. When the user communicates the need for image editing, the client devices 101-106 execute the image editing method of the embodiment of the present disclosure to determine a source image to be edited from a generated history image or a history image uploaded by the user, and edit the source image to generate a target image. The client devices 101-106 further output the generated target image to the user (e.g., output via a display) as response data (i.e., a response) for the current round of dialogue. According to some embodiments, the response data of the current round of dialogue may further include an interpretation text of the target image, which may be, for example, an explanatory text for describing the screen content of the target image, a text for describing the process and logic of generating the target image by the system, etc.

[0039] According to some embodiments, the server 120 may execute the image editing method according to the embodiment of the present disclosure. Specifically, the user can convey his / her image creation or image editing needs by operating the client devices 101-106 (for example, operating an input device such as a mouse, keyboard, or touch screen) to input an image processing command for each round of dialogue. The client devices 101-106 transmit the user's image processing command for each round of dialogue to the server 120. When the user's image processing command conveys a need for image editing, the server 120 executes the image editing method according to the embodiment of the present disclosure to determine a source image to be edited from a history image already generated in the current dialogue or a history image uploaded by the user, edits the source image to generate a target image, and outputs the target image to the client devices 101-106 as response data for the current round of dialogue. The client devices 101-106 further output the response data to the user (for example, output via a display). According to some embodiments, the response data of the current round of dialogue may further include an interpretation text of the target image, which may be, for example, an explanatory text for describing the screen content of the target image, a text for describing the process and logic of generating the target image by the system, etc.

[0040] 2 shows a flowchart of an image editing method 200 according to an embodiment of the present disclosure. As described above, the execution subject of the method 200 may be a client device such as the client devices 101-106 shown in FIG 1, or a server such as the server 120 shown in FIG 1.

[0041] As shown in FIG. 2, the method 200 includes steps S210 to S230.

[0042] In step S210, the editing command input by the user in the dialogue of the current round and the historical dialogue information in the dialogue of the historical round are obtained, where the historical dialogue information includes a historical dialogue text and at least one historical image.

[0043] In step S220, a source image to be edited is determined from at least one historical image according to the editing command and the historical dialogue information.

[0044] In step S230, the source image is edited based on the editing command to generate a target image.

[0045] According to an embodiment of the present disclosure, a multi-round interactive image editing method is provided. By combining historical interaction information, the editing object (i.e., source image) directed by the user's current editing command is accurately understood, and the source image is edited and a target image is generated according to the user's editing command. The embodiment of the present disclosure understands and meets the user's image editing needs through a unified and natural multi-round interaction manner, significantly reducing the complexity of user operation, and improving image editing efficiency and user experience.

[0046] Each step of the method 200 is described in detail below.

[0047] In step S210, the editing command input by the user in the dialogue of the current round and the history dialogue information in the dialogue of the history round are obtained.

[0048] In the embodiment of the present disclosure, a dialogue refers to an interaction process in which a user inputs a query and an AI image generation system outputs a response. According to the number of interactions between a user and an AI image generation system, the dialogue can be divided into a single-round dialogue and a multi-round dialogue. In a single-round dialogue, a user interacts with an AI image generation system only once. After a user inputs a question and receives a response output from the system, the dialogue ends. In a multi-round dialogue, a user interacts with an AI image generation multiple times. Each interaction is called a "round" of dialogue, and includes a question input by the user and a response output by the system for the question.

[0049] In an embodiment of the present disclosure, the current round of dialogue may be any round other than the first round of a multi-round dialogue, for example, the second round of dialogue, the third round of dialogue, etc.

[0050] The historical dialogue information includes a historical dialogue text and at least one historical image. The historical dialogue text includes a user input text and a system response text in a dialogue of a historical round. The at least one historical image includes a user input image and a system response image in a dialogue of a historical round.

[0051] The editing command input by the user in the current round of dialogue is used to convey the user's current image editing needs. The editing command may directly or indirectly refer to the past images. For example, the editing command may be, for example, "change the first image to a more high-tech one," "put a hat on the dog in the picture," "redraw another set of similar ones," etc.

[0052] According to some embodiments, text entered by the user in the current round of dialogue may become editing instructions.

[0053] According to some other embodiments, speech recognition may be performed on the speech input by the user in the current round of dialogue to obtain a result text, and the result text may be an editing command.

[0054] It should be explained that the user's input data in the current round of dialogue does not always convey the need for image editing, but may also convey other needs, such as image creation needs, chatting, inquiries about how to use each function of the AI ​​image generation system, etc. As can be understood, only when the user's input data conveys the need for image editing, the input data belongs to the editing command. If the user's input data conveys a need other than image editing, the input data does not belong to the editing command.

[0055] According to some embodiments, intent identification may be performed on the input data of the user's current round of interaction to determine whether the user currently has an image editing need, i.e., whether the current input data is an editing command.

[0056] Specifically, the input data of the user in the current round of dialogue and historical dialogue information in the historical round of dialogue may be obtained. Based on the input data and the historical dialogue information, the user's intention in the current round of dialogue is identified. In response to the intention being an image editing intention, the input data of the current round of dialogue is set as an editing command.

[0057] According to the above embodiment, the image editing demand of the user can be accurately identified and unnecessary image editing processes can be avoided, thereby improving the image editing efficiency and the image editing experience of the user.

[0058] According to some embodiments, a preset large-scale language model may be used to identify a user's intention. For example, input data of a user's current round of dialogue and historical dialogue information may be filled into a preset prompt template to obtain input information for the large-scale language model. The input information is input into the large-scale language model to obtain an intention identification result output from the large-scale language model.

[0059] In step S220, a source image to be edited is determined from at least one historical image according to the editing command and the historical dialogue information.

[0060] In multi-round dialogue, the editing command currently input by the user is generally related to the history dialogue content, and the editing demand may become gradually clearer during multi-round input. Therefore, it is necessary to combine the history dialogue information to accurately understand the editing object (i.e., the source image to be edited) that the user's current editing command is directed to.

[0061] According to some embodiments, in step S220, a pre-established large-scale language model may be used to determine the source image. Step S220 may include steps S221 to S223.

[0062] In step S221, a preset prompt template is obtained, which includes guide text and slots to be filled to guide the language model to determine a source image to be edited from at least one history image.

[0063] In step S222, the editing commands and historical dialogue information are filled into the slots to obtain input information.

[0064] In step S223, the input information is input to the language model to obtain a source image output from the language model. Specifically, the language model can output an identifier (ID) of the source image.

[0065] According to the above embodiment, the language understanding capability of a large-scale language model is utilized to accurately understand the image editing needs of the user, thereby ensuring the accuracy of the image editing.

[0066] According to some embodiments, a prompt template may be, for example, "User current editing command: {edit command}\nHistory dialogue text: {history dialogue text}\nHistory image: {history image}\nUser image to edit: {source image}", where {edit command}, {history dialogue text}, {history image} are slots to fill with editing command, history dialogue text, and history image, respectively, and {source image} is the output result of a large-scale language model.

[0067] According to some embodiments, before performing steps S221 to S223, high-quality annotation data of "editing command-historical dialogue information-source image" can be manually created, and the annotation data can be used to fine-tune the pre-trained large-scale language model, thereby improving the accuracy of source image identification of the large-scale language model.

[0068] According to some embodiments, a trained image language model may be utilized to determine the source image, the image language model including a text encoder and an image encoder capable of encoding text and images into the same semantic space.

[0069] Specifically, in step S220, the editing command may be encoded using a text encoder in the image language model to obtain a vector representation of the editing command. For each history image, the history image is encoded using an image encoder to obtain an initial vector representation of the history image. If there is no interpretation text in the history image, the initial vector representation is the final vector representation of the history image. If there is interpretation text in the history image, the interpretation text is further encoded using a text encoder to obtain a vector representation of the interpretation text, and the initial vector representation of the history image and the vector representation of the interpretation text are fused (e.g., summed, averaged, fused by attention mechanism, etc.) to obtain a final vector representation of the history image. The similarity (e.g., cosine similarity) between the vector representation of the editing command and the vector representation of each history image is calculated, and the history image with the largest similarity is determined as the source image to be edited.

[0070] After the source image to be edited is determined in step S220, step S230 is executed, in which the source image is edited according to the editing command to generate a target image.

[0071] According to some embodiments, step S230 may include steps S231 to S233.

[0072] In step S231, the source description text of the source image is obtained.

[0073] In step S232, a target description text of the target image is determined based on the source description text and the editing instructions.

[0074] In step S233, a target image is generated based on the target description text, where the source description text is used to provide control over the process of generating the target image.

[0075] According to the above embodiment, by using the source description text to control the generation process of the target image, it is possible to achieve the image editing effect desired by the user, on the premise that the target image and the source image are kept as similar as possible.

[0076] According to some embodiments, the explanatory text of each history image may be pre-stored, and accordingly, in step S231, the explanatory text corresponding to the source image may be obtained from the stored explanatory texts as the source explanatory text.

[0077] According to some embodiments, the source image may be an image generated by the AI ​​image generation system by invoking the text-to-image model in a dialogue of a history round, i.e., the source image is output from the text-to-image model by inputting a specified prompt text into the text-to-image model. In this case, the source prompt text is the prompt text input into the text-to-image model to generate the source image.

[0078] According to some other embodiments, the source image may be an image that a user voluntarily uploads in a historical round of dialogue, in which case, when the user uploads the image, a description text of the image, i.e., a source description text, may be generated by invoking a large-scale language model.

[0079] According to some embodiments, in step S232, a large-scale language model may be used to generate a target explanatory text. Specifically, the source explanatory text may be rewritten using the language model according to the editing command to obtain the target explanatory text. According to the embodiments, the language understanding ability and text generation ability of the large-scale language model may be used to realize intelligent rewriting, and the image editing effect may be improved by deeply understanding the user's image editing needs.

[0080] According to some embodiments, a pre-configured prompt template may be obtained for guiding the large-scale language model to generate a target explanatory text. The prompt template includes slots to be filled. The prompt template may be, for example, "Source explanatory text: {Source explanatory text}\nEditing instructions: {Editing instructions}\nTarget explanatory text: {Target explanatory text}", "Source explanatory text: {Source explanatory text}\nEditing instructions: {Editing instructions}\nHistory dialogue text: {History dialogue text}\nHistory image: {History image}\nTarget explanatory text: {Target explanatory text}", etc.

[0081] The source explanatory text and the editing instructions are filled into the corresponding slots (along with historical dialogue information depending on the needs of the prompt template), to obtain input information for the large-scale language model, which is then input into the large-scale language model to obtain a target explanatory text output from the large-scale language model.

[0082] According to some embodiments, in step S233, a predefined text-to-image diffusion model may be used to generate the target image. The text-to-image diffusion model includes a text encoder and a noise generating network. The model receives a text input and generates an image that matches the text condition.

[0083] The text-to-image diffusion model generates an image through a reverse diffusion process, which includes multiple iterations. The text-to-image diffusion model gradually removes noise from an initial image (which may be a pure noise image) through multiple iterations, and finally obtains a clear result image. In the reverse diffusion process, a given text is encoded into a high-dimensional text feature vector through a text encoder, which is used to guide a noise generation network to generate noise that matches the text condition, and the noise generated in each iteration is subtracted from the initial image in order to obtain an image that matches the text condition. Each iteration in the reverse diffusion process needs to sample the values ​​of some random variables (which are generally random noise), so each iteration has a certain randomness.

[0084] According to some embodiments, step S233 may include steps S2331 and S2332.

[0085] In step S2331, based on the source description text, a text-to-image diffusion model is utilized to denoise the first initial image by performing multiple first iterations, and random variable values ​​sampled in each first iteration of the multiple first iterations are recorded.

[0086] In step S2332, based on the target description text, a text-to-image diffusion model is used to perform multiple second iterations to denoise the second initial image to generate a target image, where each second iteration of the multiple second iterations multiplies the random variable values ​​sampled in the first iteration of the corresponding round.

[0087] According to the above embodiment, the text-to-image diffusion model is used to simulate the nose removal process (de-diffusion process) for each iteration of the source image, and estimates the random factors hidden in the source image, i.e., what random variables are sampled in each iteration when the source image is generated by the text-to-image diffusion model. These random variables contain a large amount of information about the source image.

[0088] In the process of generating the target image, the random variable values ​​sampled when generating the source image are multiplied, so that the target image can retain the details of the content of the source image, and the target image can be generated based on the target description text to achieve the editing effect expected by the user.

[0089] The text-to-image diffusion model in the above embodiment may be any existing text-to-image diffusion model. The above embodiment can achieve the editing effect expected by the user on the premise that the target image and the source image are kept as similar as possible by simply controlling the inference process of the text-to-image diffusion model. There is no need to separately create large-scale training data to train the text-to-image diffusion model, and it is possible to achieve general-purpose image editing capabilities at low cost and to have transitional capabilities between various types of text-to-image diffusion models.

[0090] It should be noted that the terms "first" and "second" in the above examples are used to distinguish between an image generation process conditioned on a source explanatory text and an image generation process conditioned on a target explanatory text. A "first iteration" refers to one iteration of an image generation process conditioned on a source explanatory text, and a "second iteration" refers to one iteration of an image generation process conditioned on a target explanatory text.

[0091] According to some embodiments, the text-to-image diffusion model includes a text encoder and a noise generating network.

[0092] According to some embodiments, in step S2331, the first initial image may be a random noise image, which may be generated by randomly sampling in a preset noise distribution (e.g., Gaussian distribution).

[0093] According to some embodiments, every first iteration in step S2331 may include the following steps S23311 to S23314.

[0094] In step S23311, the source description text is input to a text encoder to generate a source text vector corresponding to the source description text.

[0095] In step S23312, a sampled random variable value in this first iteration, i.e., an initial sampling noise, is obtained by sampling in a preset noise distribution (e.g., Gaussian distribution). The sampled random variable value may be a random variable image having the same size as the first initial image.

[0096] In step S23313, the source text vector and the sampled random variable values ​​are input into a noise generating network to obtain a predicted noise of the current image, which may be a noise image with the same size as the current image.

[0097] In step S23314, prediction noise is removed from the current image to obtain a result image for this first iteration.

[0098] As can be seen, the current image in the first iteration is the first initial image, and the current image in the second and every subsequent iteration is the resultant image generated in the previous iteration.

[0099] It should be noted that step S2331 (including steps S23311 to S23314) is used to simulate the noise removal process for each iteration of the source image, but the final image generated in the step (i.e., the result image generated in the final first iteration) is not necessarily the same as the source image.

[0100] According to some embodiments, in step S2332, a second initial image may be generated based on the source image, so that the second initial image contains information of the source image, thereby keeping the generated target image consistent with the source image as much as possible.

[0101] According to some embodiments, the second initial image may be the source image itself, whereby the second initial image retains all information of the source image.

[0102] According to some other embodiments, the second initial image may be obtained by adding noise (e.g., random noise conforming to a Gaussian distribution) to the source image, which can increase the variety and richness of the target image while keeping the overall visual effect (including color, composition, style, etc.) of the target image and the source image similar.

[0103] According to some embodiments, every second iteration in step S2332 may include steps S23321 to S23323.

[0104] In step S23321, the target explanatory text is input to a text encoder to generate a target text vector corresponding to the target explanatory text.

[0105] In step S23322, the target text vector and the random variable values ​​sampled in the first iteration of the corresponding round are input into a noise generating network to obtain the predicted noise of the current image.

[0106] In step S23323, the prediction noise is removed from the current image to obtain a result image of the current second iteration, where the current image in the first second iteration is the second initial image, and the current image in the second and subsequent second iterations is the result image generated in the previous second iteration.

[0107] According to the above embodiment, in the process of generating a target image based on the target description text, the random variable values ​​sampled when the source image was generated are multiplexed, thereby achieving the editing effect desired by the user on the premise that the target image and the source image are kept as similar as possible.

[0108] 3 shows a schematic diagram of the t-th iteration of a text-to-image diffusion model 300 according to an embodiment of the present disclosure. The text-to-image diffusion model 300 includes a text encoder 310 and a noise generation network 320. The top half of FIG. 3 shows the t-th iteration, i.e., the t-th first iteration, of the de-diffusion process conditioned on the source explanatory text t1, and the bottom half of FIG. 3 shows the t-th iteration, i.e., the t-th second iteration, of the de-diffusion process conditioned on the target explanatory text.

[0109] As shown in the upper half of FIG. 3, in the t-th iteration of the de-diffusion process conditioned on the source description text t1, the source description text t1 is input to the text encoder 310 to obtain a source text vector e1. The sampling noise y(t) of the current iteration is obtained by sampling in a preset noise distribution (e.g., Gaussian distribution). The source text vector e1 and the sampling noise y(t) are input to the noise generation network 320 to obtain a predicted noise z1(t) of the current iteration. The current image c1(t) is subtracted from the predicted noise z1(t) to obtain a result image r1(t). As can be seen, the result image r1(t) of the current iteration becomes the current image c1(t+1) of the next iteration.

[0110] As shown in the lower half of FIG. 3, in the t-th iteration of the de-diffusion process conditioned on the target description text t2, the target description text t2 is input to the text encoder 310 to obtain the target text vector e2. The sampling noise y(t) in the t-th iteration of the de-diffusion process conditioned on the source description text t1 is multiplexed, and the target text vector e2 and the sampling noise y(t) are input to the noise generation network 320 to obtain the predicted noise z2(t) of the current iteration. The current image c2(t) is subtracted from the predicted noise z2(t) to obtain the result image r2(t). As can be seen, the result image r2(t) of the current iteration becomes the current image c2(t+1) of the next iteration. After completing all T iterations, the generated result image is the target image.

[0111] According to some embodiments, in step S230, an image editing model driven by the trained linguistic instructions may be used to generate the target image, specifically, the editing instructions and the source image are input to the image editing model to obtain a target image output from the image editing model.

[0112] It should be noted that the image editing model in the above embodiment is obtained by training with a large amount of sample data of "editing command-source image-target image". Annotating the sample data and training the model requires a large amount of manpower and time cost, and has low versatility and transferability.

[0113] 4 shows a schematic diagram of an image editing process according to an embodiment of the present disclosure. The image editing process shown in FIG. 4 is realized by an AI image generation system. The AI ​​image generation system includes a context-related intent understanding module 410, a text difference-driven image editing module 420, and a history context recording module 430. When a round of dialogue is completed, the history context recording module 430 is used to store related information of the round dialogue (including dialogue text, generated image, image description, etc.) as history dialogue information.

[0114] 4, the context-related intent understanding module 410 obtains the user's dialogue input (i.e., editing command) of the current round, obtains historical dialogue information from the history context recording module 430, understands the user's image editing intent by invoking a large-scale language model, and determines the identifier (ID) of the source image to be edited and the description of the target image (i.e., the target description text). Based on the identifier of the source image, the source image and the description of the source image (i.e., the source description text) are obtained from the history context recording module 430.

[0115] The text difference driven image editing module 420 receives the source image description, the target image description, and the source image as inputs, generates a target image, and returns it to the user. The text difference driven image editing module 420 can realize the editing effect expected by the user on the premise that the generated target image and the source image are kept as similar as possible by controlling the inference process of the text-to-image diffusion model.

[0116] After the current round of interaction is completed, the target image description of the current round of the user's interaction and the generated target image are stored in the historical context recording module 430 for use in subsequent rounds of interaction.

[0117] 5 shows a schematic diagram of an example of image editing according to an embodiment of the present disclosure. As shown in FIG. 5, the editing instruction text input by the user in the dialogue of the current round is "replace oranges with tennis balls". The intent understanding module 510 first determines the image 501 (i.e., source image) to be edited from the history images generated in the current dialogue by invoking a large-scale language model, and obtains the original text (i.e., source description text) "basket full of oranges" corresponding to the image 501 to be edited.

[0118] The intent understanding module 510 utilizes a large-scale language model (i.e., a large-scale language model) to rewrite the original text “basket full of oranges” based on the current editing instruction text “replace oranges with tennis balls” to obtain a target text (i.e., a target description text) of “basket full of tennis balls.”

[0119] An image to be edited 501 , the source text “basket of oranges”, and the target text “basket of tennis balls” are input to a text difference driven image editing module 520 to generate a result image (ie, target image) 502 .

[0120] The text difference driven image editing module 520 includes a text-to-image diffusion model 521 and a diffusion process control module 522 .

[0121] The text-to-image diffusion model 521 includes a text encoder and a noise generation network. The model receives text input and generates an image that matches the text conditions.

[0122] The diffusion process control module 522 controls the randomness of the de-diffusion process of the text-to-image diffusion model 521 to maximize the similarity between the result image 502 and the image to be edited 501 and satisfy the editing needs of the user. The diffusion process control module 522 may include a randomness estimation sub-module and an image-to-image sub-module.

[0123] The randomness estimation submodule aims to inversely estimate the random factors hidden in the image to be edited. In other words, if the image to be edited is generated by the text-to-image diffusion model 521, each iteration in the inverse diffusion process is what random variables (random noise) to sample. These random variables contain a large amount of information in the image to be edited. The randomness estimation submodule starts from the random noise image and uses the original text "basket full of oranges" as a condition, simulates the noise removal process for each iteration of the image to be edited 501, and records the random variable values ​​sampled in each iteration.

[0124] The image-to-image submodule aims to generate an image that resembles the image to be edited 501 and meets the editing demands of the user. To achieve the preservation of the image to be edited, the de-diffusion process is initialized by adding noise to the image to be edited 501, thereby preserving the overall visual effects such as color and composition of the image to be edited 501, while the random variable values ​​sampled by the randomness estimation submodule are multiplexed in the de-diffusion process of the result image 502, thereby preserving more content details of the image to be edited 501. To meet the editing demands of the user and achieve the editing effect expected by the user, the text condition is replaced with the target text "basket full of tennis balls".

[0125] In some other examples, the editing instruction text input by the user in the dialogue of the current round may be "change it to an oil painting style." Through the corresponding processing of the intent understanding module 510 and the text difference driven image editing module 520, the image to be edited 501 can be determined and modified to an oil painting style to obtain the result image 503.

[0126] 6 shows a schematic diagram of a multi-round interactive image editing effect according to an embodiment of the present disclosure. In FIG. 6, U and AI are both interactions, where U represents a user and AI represents an AI image generation system (which may be called an "AI drawing assistant").

[0127] As shown in FIG. 6, in the first round of dialogue, the user inputs a natural language command 610, "Please draw a cat among flowers." The command 610 conveys the user's need for image creation. In response to the command 610, the AI ​​image generation system generates a new image 622 for the user and an interpretation text, "This is a work generated for you. Click on the picture to see a larger version." The combination of the image 622 and the interpretation text is made into a response 620 to the user command 610.

[0128] In the second round of dialogue, the user inputs a natural language command 630, "replace the cat with a dog." The command 630 conveys the user's demand for image editing. In response to the command 630, the AI ​​image generation system uses the method 200 of the embodiment of the present disclosure to determine that the image to be edited is the image 622, edits the image 622, and generates the image 642. Furthermore, by invoking a large-scale language model, an interpretation text of the image 642, "replaced with a dog. Click on the picture to see a larger view," may be generated, and the combination of the image 642 and the interpretation text may be the response 640 to the user command 630. As shown in FIG. 6, the newly generated image 642 is highly consistent with the original image 622 in terms of color, style, element (flower, animal, etc.) position, etc., and the cat in the original image 622 is replaced with a dog, satisfying the user's demand for editing.

[0129] In the third round of dialogue, the user inputs a natural language command 650, "Draw a happy expression." The command 650 conveys the user's demand for image editing. In response to the command 650, the AI ​​image generation system uses the method 200 of the embodiment of the present disclosure to determine that the image to be edited is the image 642, edits the image 642, and generates the image 662. Furthermore, by invoking a large-scale language model, an interpretation text of the image 662, "This is the work after editing. Click on the picture to see the enlarged view~", may be generated, and the combination of the image 662 and the interpretation text may be the response 660 to the user command 650. As shown in FIG. 6, the newly generated image 662 is highly consistent with the original image 642 in terms of color, style, composition, etc., and modifies the dog in the original image 642 from a sad expression to a happy expression, thereby satisfying the user's demand for editing.

[0130] In the fourth round of dialogue, the user inputs a natural language command 670, "Please remove the flower." The command 670 conveys the user's demand for image editing. In response to the command 670, the AI ​​image generation system uses the method 200 of the embodiment of the present disclosure to determine that the image to be edited is the image 662, edits the image 662, and generates an image 682. Furthermore, by invoking a large-scale language model, an interpretation text of the image 682, such as "This is a work generated for you. Click on the picture to see a larger view," may be generated, and the combination of the image 682 and the interpretation text may be used as a response 680 to the user command 670. As shown in FIG. 6, the newly generated image 682 is highly consistent with the original image 662 in terms of color, style, composition, etc., and the flower in the original image 662 has been deleted, thereby satisfying the user's demand for editing.

[0131] According to an embodiment of the present disclosure, there is further provided an image editing device. Figure 7 shows a structural block diagram of an image editing device 700 according to an embodiment of the present disclosure. As shown in Figure 7, the device 700 includes an acquisition module 710, a determination module 720 and an editing module 730.

[0132] The acquiring module 710 is configured to acquire editing commands input by a user in a current round of dialogue and historical dialogue information in a historical round of dialogue, where the historical dialogue information includes a historical dialogue text and at least one historical image.

[0133] The determining module 720 is configured to determine a source image to be edited from the at least one historical image based on the editing command and the historical interaction information.

[0134] The editing module 730 is configured to edit the source image based on the editing instructions to generate a target image.

[0135] According to an embodiment of the present disclosure, a multi-round interactive image editing device is provided. By combining historical interaction information, the editing object (i.e., source image) directed by the user's current editing command is accurately understood, and the source image is edited and a target image is generated according to the user's editing command. The embodiment of the present disclosure understands and meets the user's image editing needs through a unified and natural multi-round interaction manner, significantly reducing the complexity of user operation, and improving image editing efficiency and user experience.

[0136] According to some embodiments, the determination module includes a first acquisition unit configured to acquire a pre-set prompt template, the prompt template including guide text and slots to be filled for guiding a language model to determine a source image to be edited from the at least one historical image, a filling unit configured to fill the slots with the editing instructions and the historical dialogue information to obtain input information, and an input unit configured to input the input information to the language model to obtain the source image output from the language model.

[0137] According to some embodiments, the editing module includes a second acquisition unit configured to acquire a source description text of the source image, a determination unit configured to determine a target description text of the target image based on the source description text and the editing instructions, and a generation unit configured to generate the target image based on the target description text, wherein the source description text is used to control the process of generating the target image.

[0138] According to some embodiments, the determining unit is further configured to rewrite the source explanatory text using a language model based on the editing instructions to obtain the target explanatory text.

[0139] According to some embodiments, the generating unit includes a simulating subunit configured to denoise a first initial image by performing a plurality of first iterations using a text-to-image diffusion model based on the source explanatory text and record a random variable value sampled in each first iteration of the plurality of first iterations, and a generating subunit configured to denoise a second initial image by performing a plurality of second iterations using the text-to-image diffusion model based on the target explanatory text to generate the target image, where each second iteration of the plurality of second iterations multiplexes the random variable value sampled in the first iteration of the corresponding round.

[0140] According to some embodiments, the text-to-image diffusion model includes a text encoder and a noise generation network, and wherein each second iteration of the multiple second iterations includes: inputting the target description text to the text encoder to generate a target text vector corresponding to the target description text; inputting the target text vector and random variable values ​​sampled in a first iteration of a corresponding round to the noise generation network to obtain predicted noise for a current image; and removing the predicted noise from the current image to obtain a result image of the current second iteration, wherein the current image in a first second iteration is the second initial image, and the current image in a second and each subsequent second iteration is a result image generated in a previous second iteration.

[0141] According to some embodiments, the second initial image is generated based on the source image.

[0142] According to some embodiments, the second initial image is obtained by adding noise to the source image.

[0143] It should be understood that each module and unit of the apparatus 700 shown in Fig. 7 may correspond to each step in the method 200 described with reference to Fig. 2. Thus, the operations, features and advantages described with respect to the method 200 are equally applicable to the apparatus 700 and the modules and units included therein. For the sake of brevity, some operations, features and advantages will not be described herein.

[0144] Although particular functionality has been discussed above with reference to particular modules, it should be noted that the functionality of each module discussed herein may be split into multiple modules and / or at least some of the functionality of multiple modules may be combined into a single module.

[0145] It should be further understood that the present specification may describe various techniques in the general context of software and hardware elements or program modules. Each unit described with respect to FIG. 7 may be implemented in hardware or hardware combining software and / or firmware. For example, the units may be implemented as computer program code / instructions configured to be executed on one or more processors and stored on a computer-readable storage medium. Alternatively, the units may be implemented as hardware logic / circuitry. For example, in some embodiments, one or more of the modules 710-730 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (e.g., a processor (e.g., including a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and / or one or more components in other circuits) that may optionally execute received program code and / or include embedded firmware to perform functions.

[0146] According to an embodiment of the present disclosure, there is further provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform an image editing method of an embodiment of the present disclosure.

[0147] According to an embodiment of the present disclosure, there is further provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the image editing method of the embodiment of the present disclosure.

[0148] According to an embodiment of the present disclosure, there is further provided a computer program product including computer program instructions, which, when executed by a processor, implements the image editing method of the embodiment of the present disclosure.

[0149] With reference to FIG. 8, a structural block diagram of an electronic device 800 that can be used as a server or client of the present disclosure is described, which is an example of a hardware device that can be applied to each aspect of the present disclosure. The electronic device represents various types of digital electronic computing devices, such as laptop computers, desktop computers, stages, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various types of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components, their connections, and their functions shown herein are merely exemplary and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] 8, the electronic device 800 includes a computing unit 801, which can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 802 or loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may further store various programs and data required for the operation of the electronic device 800. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0151] A number of components in the electronic device 800 are connected to an I / O interface 805, and include an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input numeric or character information and generate key signal input for user settings and / or function control of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track board, a track ball, a control lever, a microphone, and / or a remote control. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 enables the electronic device 800 to exchange information / data with other devices via a computer network, e.g., the Internet, and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, e.g., a Bluetooth device, an 802.11 device, a Wi-Fi device, a WiMAX device, a cellular communication device, and / or the like.

[0152] The computing unit 801 may be any of a variety of general-purpose and / or special-purpose processing assemblies having processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 executes each of the methods and processes described above, such as method 200. For example, in some embodiments, the method 200 may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, some or all of the computer program may be loaded and / or installed in the electronic device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, it may perform one or more steps of the method 200 described above. Optionally, in other embodiments, the computing unit 801 may be configured to perform the method 200 in any other suitable manner (eg, by firmware).

[0153] Various embodiments of the systems and techniques described herein above may be implemented in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: embodied in one or more computer programs that may be executed and / or interpreted by a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor, and may receive data and instructions from a storage system, at least one input device, at least one output device, and may transmit data and instructions to the storage system, the at least one input device, and the at least one output device;

[0154] The program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, a special purpose computer or other programmable data processing apparatus, so that when the program codes are executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program codes may be entirely executed on a machine, partly executed on a machine, partly executed on a machine and partly executed on a remote machine as a separate software package, or entirely executed on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in combination with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more wire-based electrical connections, portable computer disks, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0156] To provide for interaction with a user, the systems and techniques described herein may be implemented in a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user may provide input to the computer. Other types of devices may also be provided for providing interaction with a user, for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form (including audio input, voice input, tactile input).

[0157] The systems and techniques described herein may be implemented in a computing system including a background component (e.g., a data server), a middleware component (e.g., an application server), a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or any combination of such background, middleware, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0158] The computer system may include a client and a server. The client and the server are generally remote from each other and usually interact with each other via a communication network. The client-server relationship is created by running computer programs in the corresponding computers that have a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server incorporating a blockchain.

[0159] It should be understood that steps may be reordered, added, or removed using the various types of flows shown above. For example, each step described in this disclosure may be performed in parallel, sequentially, or in a different order, and is not limited herein, as long as the technical solution disclosed in this disclosure can achieve the desired results.

[0160] Although the embodiments or examples of the present disclosure have been described with reference to the drawings, it should be understood that the above methods, systems, and devices are merely exemplary embodiments or examples, and the scope of the present disclosure is not limited by these embodiments or examples, but only by the scope of the claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalent elements. In addition, each step may be performed in a different order from the order described in this disclosure. Furthermore, various elements in the embodiments or examples may be combined in various ways. In essence, with the evolution of technology, many elements described herein may be replaced by equivalent elements appearing later in this disclosure.

Claims

1. 1. An image editing method, comprising: Acquiring an editing command input by a user in a dialogue of a current round and historical dialogue information in a dialogue of a historical round, the historical dialogue information including a historical dialogue text and at least one historical image; determining a source image to be edited from the at least one historical image based on the editing command and the historical dialogue information; and editing the source image based on the editing instructions to generate a target image.

2. determining a source image to be edited from the at least one historical image based on the editing command and the historical dialogue information; obtaining a preset prompt template, the prompt template including guide text and slots to be filled for guiding a language model to determine a source image to be edited from the at least one historical image; filling said slots with said editing instructions and said historical dialogue information to obtain input information; The method of claim 1 , further comprising inputting the input information into the language model to obtain the source image as an output from the language model.

3. The step of editing the source image based on the editing command includes: obtaining a source description text of the source image; determining a target description text for the target image based on the source description text and the editing instructions; 2. The method of claim 1, further comprising: generating the target image based on the target description text, the source description text being used to provide control over the process of generating the target image.

4. determining a target description text for the target image based on the source description text and the editing instructions includes: The method of claim 3 , further comprising: rewriting the source explanatory text utilizing a language model based on the editing instructions to obtain the target explanatory text.

5. generating the target image based on the target description text includes: denoising a first initial image by performing a plurality of first iterations utilizing a text-to-image diffusion model based on the source description text, and recording a random variable value sampled in each of the plurality of first iterations; and performing a second iteration utilizing the text-to-image diffusion model based on the target description text to denoise the second initial image to generate the target image; 4. The method of claim 3, wherein each second iteration of the plurality of second iterations multiplexes the random variable values ​​sampled in a first iteration of the round.

6. The text-to-image diffusion model includes a text encoder and a noise generating network, and wherein each second iteration of the plurality of second iterations comprises: inputting the target descriptive text into the text encoder to generate a target text vector corresponding to the target descriptive text; inputting the target text vector and the random variable values ​​sampled in the first iteration of the corresponding round into the noise generating network to obtain a predicted noise for the current image; removing the prediction noise from the current image to obtain a result image of this second iteration; 6. The method of claim 5, wherein the current image in the first second iteration is the second initial image, and the current image in the second and each subsequent second iteration is a resultant image generated in the previous second iteration.

7. The method of claim 5 , wherein the second initial image is generated based on the source image.

8. The method of claim 7 , wherein the second initial image is obtained by adding noise to the source image.

9. An image editing device, comprising: an acquisition module configured to acquire an editing command input by a user in a dialogue of a current round and historical dialogue information in a dialogue of a historical round, the historical dialogue information including a historical dialogue text and at least one historical image; a determining module configured to determine a source image to be edited from the at least one historical image based on the editing command and the historical interaction information; an editing module configured to edit the source image based on the editing instructions to generate a target image.

10. The determination module includes: a first acquisition unit configured to acquire a preset prompt template, the prompt template including guide text for guiding a language model to determine a source image to be edited from the at least one historical image and slots to be filled; a filling unit configured to fill the slots with the editing instructions and the historical dialogue information to obtain input information; and an input unit configured to input the input information into the language model to obtain the source image output from the language model.

11. The editing module includes: a second acquisition unit configured to acquire a source description text of the source image; a determining unit configured to determine a target description text of the target image based on the source description text and the editing instructions; The apparatus of claim 9 , further comprising: a generation unit configured to generate the target image based on the target description text, the source description text being used to provide control over a process of generating the target image.

12. The determination unit further comprises: The apparatus of claim 11 , configured to rewrite the source explanatory text utilizing a language model based on the editing instructions to obtain the target explanatory text.

13. The generating unit comprises: a simulating subunit configured to denoise a first initial image by performing a plurality of first iterations utilizing a text-to-image diffusion model based on the source description text, and record a random variable value sampled in each of the plurality of first iterations; a generating subunit configured to denoise a second initial image by performing a second iteration using the text-to-image diffusion model based on the target description text to generate the target image; The apparatus of claim 11 , wherein each second iteration of the plurality of second iterations multiplexes the random variable values ​​sampled in a first iteration of the round.

14. The text-to-image diffusion model includes a text encoder and a noise generating network, and wherein each second iteration of the plurality of second iterations comprises: inputting the target descriptive text into the text encoder to generate a target text vector corresponding to the target descriptive text; inputting the target text vector and the random variable values ​​sampled in the first iteration of the corresponding round into the noise generating network to obtain a predicted noise for the current image; removing the prediction noise from the current image to obtain a result image of this second iteration; 14. The apparatus of claim 13, wherein the current image in a first second iteration is the second initial image, and the current image in a second and each subsequent second iteration is a resultant image generated in a previous second iteration.

15. The apparatus of claim 13 , wherein the second initial image is generated based on the source image.

16. The apparatus of claim 15 , wherein the second initial image is obtained by adding noise to the source image.

17. At least one processor; and a memory communicatively coupled to the at least one processor, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

18. A non-transitory computer readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the method of any one of claims 1 to 8.

19. A computer program product comprising computer program instructions which, when executed by a processor, implements the method of any one of claims 1 to 8.

Images