Method and device for generating image from text, electronic equipment and storage medium

Abstract

The invention provides a method and device for generating an image from a text, electronic equipment and a storage medium, and relates to the field of artificial intelligence. The method for generating the image from the text comprises the following steps: acquiring an original image, a source text description corresponding to the original image and a target text description; encoding the source text description to obtain a first text embedding representation, and encoding the target text description to obtain a second text embedding representation; according to a pre-trained first text generation image model, updating the second text embedding representation to obtain a third text embedding representation; adjusting parameters of the first text generation image model according to the third text embedding representation to obtain a second text generation image model; and generating an image model according to the second text embedding representation and the second text to obtain a target image. According to the method, editing of the image can be controlled for the target text while key information of the original image is reserved, the modification requirement of a user for the image is met, and the use threshold of the user is lowered.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for generating images from text. Background Art

[0002] Image editing has always been a hot topic in the field of computer vision. Text-based image generation (i.e., text-generated image) technology has broad application prospects in many scenarios, including the creation of personalized wallpapers for mobile phone themes, acquisition of creative image materials for slideshows, content creation in virtual spaces, and multi-modal dialogue interaction systems. However, the related technologies require a lot of input materials when generating images from text, and the user threshold is high. Summary of the invention

[0003] The present application provides a method, device, electronic device and storage medium for generating images from text, which can retain the key information of the original image while controlling the editing of the image for the target text, meet the user's needs for image modification, and lower the user's usage threshold.

[0004] In a first aspect, an embodiment of the present application provides a method for generating an image from text, comprising:

[0005] Obtaining an original image, a source text description corresponding to the original image, and a target text description;

[0006] The source text description is encoded to obtain a first text embedding representation, and the target text description is encoded to obtain a second text embedding representation; wherein the source text description is used to describe the picture content of the original image, and the target text description is used to describe the picture content of the target image generated by image editing the original image;

[0007] Inputting the original image, the first text embedding representation and the second text embedding representation into a pre-trained first text generation image model, wherein the first text generation image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and updating the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image;

[0008] Inputting the original image and the third text embedding representation into the first text-to-image generation model, adjusting the parameters of the first text-to-image generation model, and obtaining a second text-to-image generation model;

[0009] An image model is generated according to the second text embedding representation and the second text to obtain a target image.

[0010] In a second aspect, an embodiment of the present application provides a device for generating an image from text, characterized in that it includes:

[0011] A first acquisition unit, used to acquire an original image, a source text description corresponding to the original image, and a target text description;

[0012] An encoding unit, configured to encode the source text description to obtain a first text embedding representation, and to encode the target text description to obtain a second text embedding representation; wherein the source text description is used to describe the picture content of the original image, and the target text description is used to describe the picture content of the target image generated by image editing the original image;

[0013] A pre-trained first text-generated image model is used to input the original image, the first text embedding representation and the second text embedding representation, the first text-generated image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and to update the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image;

[0014] an adjustment unit, configured to input the original image and the third text embedding representation into the first text-generated image model, and adjust parameters of the first text-generated image model to obtain a second text-generated image model;

[0015] The second acquisition unit is used to generate an image model according to the second text embedding representation and the second text to obtain a target image.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method in the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method in the first aspect.

[0019] In a sixth aspect, an embodiment of the present application provides a computer program, which enables a computer to execute the method in the first aspect.

[0020] The embodiment of the present application updates the second text embedding representation corresponding to the target text description according to the first text embedding representation of the source text description corresponding to the original image and the original image to obtain a third text embedding representation, so that the third text embedding representation can contain the spatial information of the original image, and further adjusts the parameters of the first text generation image model according to the third text embedding representation to obtain the second text generation image model, and the second text generation image model can restore the original image according to the third text embedding representation, so that according to the second text embedding representation and the second text generation model, the target image that retains the key information of the original image can be obtained with the second text embedding representation as a text condition.

[0021] Therefore, the embodiments of the present application can fine-tune the pre-trained text-generated image model through the original image, the source text description corresponding to the original image, and the target text description, so that the model can control the editing of the picture according to the text to obtain the target image, and the target image can retain the key information of the original image, so that only text is needed without additional input conditions such as image masks or multiple input images, so as to control the editing of the original image, and retain the key information of the original image while controlling the editing of the image for the target text, thereby meeting the user's needs for image modification and lowering the user's usage threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of an application scenario of the solution of an embodiment of the present application;

[0023] Figure 2 A schematic diagram of the interface for searching for emoticons;

[0024] Figure 3 A schematic flow chart of a method for generating an image from text according to an embodiment of the present application;

[0025] Figure 4 is a specific example of an original image and a target image according to an embodiment of the present application;

[0026] Figure 5 is a schematic diagram of the LDM structure;

[0027] Figure 6 A schematic flow chart of another method for generating an image from text according to an embodiment of the present application;

[0028] Figure 7 is a schematic diagram of a network architecture according to an embodiment of the present application;

[0029] Figure 8 A schematic diagram of the process of fusing pixel features and text embedding representations in the attention layer;

[0030] Fig. 9A A schematic diagram of the result of fusing pixel features and text embedding representation using an attention layer;

[0031] Fig. 9B A schematic diagram of the attention map corresponding to each timestamp;

[0032] Fig. 10A is a schematic diagram of obtaining a third attention map according to an embodiment of the present application;

[0033] Fig. 10B A schematic diagram of an image style conversion effect obtained according to an embodiment of the present application;

[0034] Fig.11 A schematic flow chart of another method for generating an image from text according to an embodiment of the present application;

[0035] Fig.12 is a schematic diagram of another network architecture according to an embodiment of the present application;

[0036] Fig.13 is a schematic diagram of another network architecture according to an embodiment of the present application;

[0037] Fig.14 A schematic diagram of the effect of an image obtained according to an embodiment of the present application;

[0038] Fig.15 A schematic block diagram of a device for generating an image from text according to an embodiment of the present application;

[0039] Fig.16 is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0040] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0041] It should be understood that in the embodiment of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based only on A, and B can also be determined based on A and / or other information.

[0042] In the description of the present application, unless otherwise specified, "at least one" means one or more, and "plurality" means two or more than two. In addition, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0043] It should also be understood that the first, second, etc. descriptions appearing in the embodiments of the present application are only used for illustration and distinction of the description objects, without any distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0044] It should also be understood that the specific features, structures or characteristics related to the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner.

[0045] In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or device.

[0046] The embodiments of the present application are applied to the field of artificial intelligence technology.

[0047] Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0048] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0049] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. I believe that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0050] The embodiments of the present application may involve natural language processing (NLP) in artificial intelligence technology, which is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language used by people in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes technologies such as text processing, semantic understanding, machine translation, robot question answering, and knowledge graphs.

[0051] The embodiments of the present application may involve computer vision (CV) technology in artificial intelligence technology. Computer vision is a science that studies how to make machines "see". To put it more concretely, it refers to machine vision such as using cameras and computers to replace human eyes to identify, monitor and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0052] The embodiments of the present application may also involve machine learning (ML) in artificial intelligence technology. ML is a multi-disciplinary cross-discipline involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning generally include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0053] The following is a description of the relevant terms involved in the embodiments of the present application.

[0054] Pre-training: A process of training a machine learning network model using a large data set so that the machine learning network model learns the common features in the data set. The machine learning model is independent of the specific task. The purpose of pre-training is to provide high-quality model parameters for subsequent model training on a specific data set. In the embodiment of the present application, the training data can be text, text-image pairs, text-video pairs, etc., and the machine learning model can be a text-to-image model, such as a diffusion model, a potential diffusion model, etc. The pre-trained model can use self-supervised learning techniques, such as autoregressive language models and autoencoding techniques.

[0055] Fine-tuning: A process of further training a pre-trained machine learning model using a specific data set. Fine-tuning is to modify the network model for a specific task. In the embodiment of the present application, the pre-trained text-to-image model can be used to support text-to-image technology after fine-tuning.

[0056] At present, when generating images from text, related technologies require additional inputs based on the input image, such as inputting an image mask to indicate the area in the image that needs to be edited, inputting multiple images of the same object to learn the shape of the target object, and a description of the original input image to summarize the content of the input image. Related technologies require the use of more input materials and have a high threshold for use.

[0057] In order to solve the above technical problems, the present application provides a method, device, electronic device and storage medium for generating images from text, by updating the second text embedding representation corresponding to the target text description according to the first text embedding representation of the source text description corresponding to the original image and the original image to obtain a third text embedding representation, so that the third text embedding representation can contain the spatial information of the original image, and further adjusting the parameters of the first text-generated image model according to the third text embedding representation to obtain a second text-generated image model, and the second text-generated image model can restore the original image according to the third text embedding representation, so that according to the second text embedding representation and the second text generation model, the target image that retains the key information of the original image can be obtained with the second text embedding representation as a text condition.

[0058] Therefore, the embodiments of the present application can fine-tune the pre-trained text-generated image model through the original image, the source text description corresponding to the original image, and the target text description, so that the model can control the editing of the picture according to the text to obtain the target image, and the target image can retain the key information of the original image, so that only text is needed without additional input conditions such as image masks or multiple input images, so as to control the editing of the original image, and retain the key information of the original image while controlling the editing of the image for the target text, thereby meeting the user's needs for image modification and lowering the user's usage threshold.

[0059] In addition, the method of generating images from text provided in the embodiment of the present application has strong generalization. For example, it can support a variety of image editing operations, such as painting over the image, adding an object, style transfer, etc., that is, one model can complete multiple tasks and has a wide range of usage scenarios.

[0060] Figure 1 A schematic diagram showing an application scenario of an embodiment of the present application.

[0061] like Figure 1 As shown, the application scenario involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network. The server 1 can be a background server of the terminal device 2.

[0062] Exemplarily, the terminal device 2 may refer to a type of device that has a rich human-computer interaction mode, has the ability to access the Internet, is usually equipped with various operating systems, and has strong processing capabilities. The terminal device may be a smart phone, a tablet computer, a portable laptop, a desktop computer, a wearable device, a vehicle-mounted device, and other terminal devices, but is not limited thereto. Optionally, in the embodiment of the present application, the terminal device 2 is installed with an image editing application or an application with an image editing function.

[0063] Server 1 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can also become a node of the blockchain.

[0064] The server may be one or more servers. When there are multiple servers, there are at least two servers for providing different services, and / or there are at least two servers for providing the same service, such as providing the same service in a load balancing manner, which is not limited in the embodiments of the present application.

[0065] The terminal device and the server can be directly or indirectly connected by wired or wireless communication, and this application does not limit this. This application does not limit the number of servers or terminal devices. The solution provided by this application can be completed independently by the terminal device, or by the server, or by the terminal device and the server in cooperation, and this application does not limit this.

[0066] It should be understood that Figure 1 This is only an exemplary description and does not specifically limit the application scenarios of the embodiments of the present application. Figure 1 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and this application does not impose any limitation on this.

[0067] The embodiments of the present application can be applied to any application scenario of text-guided image editing, including but not limited to picture search, emoticon search, emoticon association, etc. Figure 2A schematic diagram of the interface for expression search is shown. Figures (a) to (c) all use text-generated image models to generate expressions based on the input text. As shown in Figure (a), when text is entered in the tag search box and an expression icon is selected, the text-generated image model can control the editing of the image based on the input text to obtain and display the target expression. As shown in Figure (b), when "text 1" is entered in the text input box through the input keyboard, the text-generated image model can control the editing of the image based on text 1 to obtain the target expression, such as expression 1, expression 2, expression 3, and expression 4. As shown in Figure (c), when text is entered in the tag search box, such as selecting any one of text 2 to text 7, or entering other text, the text-generated image model can control the editing of the image based on the input text to obtain the target expression. Among them, text 2 to text 7 can be historical search texts, and expression 5 to expression 8 can be historical target expressions.

[0068] Exemplarily, the text-generated image model can control the editing of the original expression image to generate a target expression based on the input text. The original expression image can be an expression image saved in the expression library, or any expression image input by the user, without limitation. Optionally, the expression library can also store text descriptions corresponding to each expression. Specifically, based on the embedded representation of the original expression image and the text description corresponding to the original expression image, the embedded representation of the input text can be updated, and the updated text embedded representation can contain the spatial information of the original expression. Further, the parameters of the text-generated image model are adjusted based on the updated text embedded representation, so that the adjusted text-generated image can restore the original expression according to the updated text embedded representation, so that the target expression that retains the key information of the original expression can be obtained by inputting text into the adjusted text-generated image.

[0069] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0070] Figure 3 is a schematic flow chart of a method 300 for generating an image from text according to an embodiment of the present application. The method 300 can be executed by any electronic device having data processing capability. For example, the electronic device can be implemented as a server or a terminal device. Figure 1 The server 1 or terminal device 2 in the present application is not limited to this.

[0071] like Figure 3 As shown, method 300 includes steps 310 to 350 .

[0072] 310, obtaining an original image, a source text description corresponding to the original image, and a target text description.

[0073] The original image is the input image (inputimage), i.e., the image to be edited. The source text description (sourceprompt) corresponding to the original image is used to describe the screen content of the original image. The target text description (targetprompt) is used to describe the screen content of the target image generated by image editing of the original image. In an embodiment of the present application, the original image is edited according to the target text description, and the resulting image is the target image, which can retain the key information of the original image.

[0074] As a specific example, Figure 4 A specific example of the original image and the target image is shown. Figure 4 For the original image shown in (a), the source text description can be "a bird standing". When the target image shown in (b) is expected, the target text description can be "a bird spreading wings". The target image retains the appearance of the bird in the original image, but has a different posture from the bird in the original image. Here, the appearance of the bird is a specific example of the key information of the original image.

[0075] In some embodiments, the original image may be processed to obtain a source text description corresponding to the original image. The image processing process includes but is not limited to image recognition, image semantic understanding, etc., which is not limited in this application. Optionally, in other embodiments, the user may also manually input a source text description corresponding to the original image.

[0076] 320, encode the source text description to obtain a first text embedding representation, and encode the target text description to obtain a second text embedding representation.

[0077] For example, the source text description can be input into a text encoder (textEncoder), and the output is a first text embedding representation (Embedding), which is the source text embedding representation (e soc ); Input the target text description into the text encoder (textEncoder), and output the second text embedding representation, which is the target text embedding representation (e tgt ).

[0078] 330, input the original image, the first text embedding representation and the second text embedding representation into a pre-trained first text generation image model, the first text generation image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and update the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image.

[0079] Specifically, in step 330, the generated image (i.e., model output) is obtained by using the first text embedding representation and the second text embedding representation as text conditions, and then the loss is calculated and adjusted by comparing the original image and the model output. Because the second text embedding representation is optimized using the loss between the original image and the model output as the objective function during the training process of step 330, the model output gradually approaches the original image during the optimization process, and in this process, the third text embedding representation also gradually includes the spatial information of the original image, such as spatial layout information and geometric shape information.

[0080] In some embodiments, the third text embedding is represented as a vector that is closest to the second text embedding. That is, the third text embedding is represented as a vector that is closest to the semantics of the target text description and contains the spatial information of the original image. As a specific example, for Figure 4 For the input image in , the third text embedding representation can be an embedding representation vector that is closest to the text embedding representation vector of “abirdspreadingwings” and has semantic information about the standing posture of the bird in the original image.

[0081] In some embodiments, the first text generation model can integrate the first text embedding representation and the second text embedding representation into the generation process of the original image through an attention mechanism. Specifically, inputting the first text embedding representation and the pixel feature into the attention mechanism can obtain an attention map for each input token of the first text embedding representation, and the attention map contains the spatial information of the original image. Among them, the pixel feature can be the feature of the image obtained by diffusion processing the input original image. The attention mechanism can further control the generation of the image according to the attention map and the second text embedding representation to obtain the generated image (i.e., the model output). The processing process of the attention mechanism can be seen below. Figure 8 Furthermore, by optimizing the second text embedding representation, the model output can gradually approach the original image, and the optimized second text embedding representation (i.e., the third text embedding representation) gradually corresponds to the above attention map. Since the attention map retains the control information of the original image, the third text embedding representation updated according to the attention map can include the spatial information of the original image.

[0082] Exemplarily, the text-generated image model involved in the present application includes a stable diffusion model, which can also be called a latent diffusion model (LDM). The text-generated image model is described as LDM as an example. Exemplarily, the first text-generated image model is a pre-trained LDM, and the second text-generated image model is a fine-tuned LDM.

[0083] First combine Figure 5 The principle of LDM is introduced.

[0084] Figure 5 FIG. 4 shows a schematic diagram of an LDM structure, which includes a trained autoencoder model (autoEncoder), including an encoder E and a decoder D. Figure 5 As shown, the encoder E is used to compress the image x in the pixel space (PixelSpace) to obtain the image z of the image x in the latent space (LatentSpace), and then diffusion processing (DiffusionProcess) is performed on z in the latent space to obtain the random noise image z T , and then through T denoising U-type networks (DenoisingUNet)∈ θ For a random noise image z T De-noising is performed to obtain z, and finally the decoder D is used to restore z to the original pixel space to obtain the restored image

[0085] Among them, image z is obtained by diffusion processing of image z T The process can be called forward diffusion processing, and the corresponding processing module can be a diffusion module; T The process of performing denoising to obtain the image z may be referred to as a reverse diffusion process, and the corresponding processing module may be a denoising module. Figure 5 As shown, the denoising module may include T denoising Unet∈ θ .

[0086] First, let’s briefly introduce the common diffusion model (DM), which can be interpreted as a time series denoising autoencoder ∈ θ ( T ,); t = 1…T, its goal is to T Predict a corresponding denoised variant, where x T is the noisy version of the input image x, and the corresponding objective function L DM It can be expressed as follows:

[0087]

[0088] Where time t is uniformly sampled from {1,…,T}; E x,∈～N(0,), represents the expectation, ∈~N(0,1) represents the noise term randomly sampled from the (0,1) normal distribution, ∈ is the prior space, ∈ θ (x t ,) represents the output of the model.

[0089] In LDM, an autoencoder model is introduced, so that the encoder can be used to encode the image x in the pixel space to obtain the image z, so that the model can learn in the latent space. The corresponding objective function can be expressed as follows:

[0090]

[0091] The above process is used for unconditional image generation. LDM can also be used for conditional image generation, which can be extended to obtain the conditional temporal denoising autoencoder ∈ θ ( T ,,) to achieve this, so that the image generation process can be controlled by condition y. Specifically, Figure 5 As shown, this can be achieved by adding a cross-attention mechanism to the U-net network. θ ( T ,,). In order to be able to preprocess y from multiple different modalities, a domain-specific encoder τ can be introduced θ , to map y into an intermediate representation This makes it easy to introduce various forms of conditions, such as the semantic map, text, descriptions, images, or categories and layouts not shown in the figure. The final LDM model integrates the conditional control information into the middle layer of Unet through the attention layer (cross-attention, represented as QKV). The implementation of the attention layer is as follows:

[0092]

[0093]

[0094] Among them, d is the dimension of input information, K T represents the transpose of K, is an intermediate representation of Unet, Represent weights respectively.

[0095] The corresponding objective function can be expressed as follows:

[0096]

[0097] Among them, E z,y,∈～N(0,), represents expectation, ∈ θ ( t ,, θ (y)) represents the output of the model.

[0098] In some embodiments, see Figure 6 When the first text generation image model includes a diffusion module and a denoising module, step 330 can be specifically implemented by following steps 331 to 334:

[0099] 331, adding noise to the original image and inputting the image into a diffusion module to obtain a first random noise.

[0100] See also Figure 7 , the original image may be the input image 701 in the figure, noise may be added to the input image 701 and then input into the first LDM 702, the first LDM 702 is a pre-trained latent diffusion model (pre-TrainedLDM). Figure 5 As an example, the encoder E in the model first encodes the input image 701 (corresponding to x), maps the input image 701 from the pixel space to the latent space (corresponding to z), and then the diffusion module can diffuse the input image with added noise to obtain the first random noise corresponding to the input image (corresponding to z). T ).

[0101] 332, input the first text embedding representation, the second text embedding representation and the first random noise into a denoising module, and obtain a first generated image using the first text embedding representation and the second text embedding representation as text conditions.

[0102] For example, see Figure 7 , the source text description corresponding to the input image 701 can be "abirdstanding", and the corresponding embedding representation can be recorded as e sou (The first example of the first text embedding representation), the target text description can be "abirdspreadingwings", and the corresponding embedding representation can be recorded as e tgt (The first example of the second text embedding representation). Taking the first LDM702 as Figure 5 As an example, the structure shown in the figure, the source text description embedding representation e soc and the embedding representation e of the target text description tgtThe first random noise can be input into the denoising module in the first LDM 702 , and the first text embedding representation and the second text embedding representation can be used as text conditions to perform reverse diffusion processing to obtain a first generated image 703 .

[0103] In some embodiments, the denoising module in the first LDM 702 includes an attention layer. Figure 5 As an example of the structure shown in the figure, the denoising module may include a denoising Unet∈ θ ,Unet∈ θ The intermediate layers of can include attention layers, denoted as QKV.

[0104] Figure 8 A schematic diagram shows the process of attention layer fusing pixel features and text embedding representation. Figure 8 As shown, pixel features It can be used as the query (Query, Q) input of the attention layer, and the text embedding representation can be used as the key (Key, K) input of the attention layer. According to the input Q and V, the attention map corresponding to each text input token can be obtained. The attention map can also be called the attention matrix, which is not limited in this application. The text embedding representation can also be used as the value (Value, V) input of the attention layer, and the output is obtained by applying it to the attention map.

[0105] Fig. 9A A schematic diagram showing the result of fusing pixel features and text embedding representation using an attention layer. The leftmost figure (a) is a visually synthesized image, and its corresponding pixel features can be input into the attention layer as Q values. The text description corresponding to the synthesized image is "afurrybearwatchingabird", and its text embedding representation can be input into the attention layer as K values ​​and V values. Figures (b) to (g) in the figure are the attention maps corresponding to each input token of the text embedding representation, for example, figure (b) is the attention map corresponding to the input token "a" of the text embedding representation, and figure (c) is the attention map corresponding to the input token "furry" of the text embedding representation, and so on.

[0106] Fig. 9BA schematic diagram of the attention map corresponding to each timestamp is shown. The first row of the input token corresponding to "bear" corresponds to the attention map of each timestamp during the iteration from t=T to t=1, and the second row of the input token corresponding to "bird" corresponds to the attention map of each timestamp during the iteration from t=T to t=1. Taking the first row of attention maps as an example, the input token corresponding to "bear" corresponds to random noise at t=T. In the process of iteration to t=1, the attention map gradually contains information such as the spatial layout and geometric shape of "bear".

[0107] It can be seen that the attention map of the input image can represent the spatial information of the image, such as spatial layout and geometric shape, and for image editing tasks, the attention map can be controlled by text to change the spatial layout and geometric shape of the image. Based on this, the embodiment of the present application obtains the first generated image through the following steps 3321 to 3324 to optimize the second text embedding representation.

[0108] 3321, input the first text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a first attention map, which includes an attention map corresponding to each text input of the first text embedding representation.

[0109] For example, Figure 8 For example, the first random noise can be used as the pixel feature The first text embedding representation is used as the text embedding representation, and is input into the attention layer as the Q value and the K value respectively, to obtain an attention map corresponding to each text input of the first text embedding representation. The attention map can be called a source attention map. The attention map includes spatial information of the original image, such as spatial layout and geometric shape information.

[0110] 3322, input the second text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a second attention map, which includes an attention map corresponding to each text input of the second text embedding representation.

[0111] For example, Figure 8 For example, the first random noise can be used as the pixel feature The second text embedding representation is used as the text embedding representation, and is input into the attention layer as the Q value and the K value respectively, to obtain an attention map corresponding to each text input of the second text embedding representation. The attention map can be called a target attention map. The attention map includes spatial information of an image (such as a target image) corresponding to the target text description, such as spatial layout and geometric shape information.

[0112] 3323, obtain a third attention map according to the first attention map and the second attention map.

[0113] Here, how to obtain the third attention map based on the first attention map and the second attention map can be determined according to the change of the target text description relative to the source text description.

[0114] As an implementation method, if the description words in the source text description are replaced to obtain the target text description, the first attention map is determined as the third attention map. Fig. 10A In (a), when a word in the source text description is replaced, the sourcemap M corresponding to the source text description can be used. * Replace target mapM t , retaining the original spatial information of the original image. It should be understood that the description words in the source text description are replaced to obtain the target text description, which can correspond to the case where the object in the original image is replaced with the target object to obtain the target image.

[0115] As another possible implementation, if the target text description is obtained by adding a description word to the source text description, then the first attention map is inserted into the second attention map according to each text input position represented by the first text embedding to obtain a third attention map. Fig. 10A In (b), when additional words are added to the source text description, the sourcemapM corresponding to the source text description can be mapped according to the position of each text input token corresponding to the source text description. * Insert target mapM t , and get the third attention map, denoted as The original spatial information of the original image is retained. It should be understood that adding description words to the source text description to obtain the target text description can correspond to adding objects to the original image to obtain the target image.

[0116] Optional, such as Fig. 10A As shown in Figure (c), the weight of at least one of the attention maps in the third attention map can also be adjusted. By adjusting the weight of the attention map, the target image can be affected, such as the style of the image. Exemplarily, by adjusting the weight of each attention map in the third attention map, the style of the image can be transferred. As a specific example, Fig. 10B As shown, by adjusting the weight of each attention map in the third attention map, a realistic waterfall image can be converted into a hand-painted waterfall image.

[0117] 3324. Using the second text embedding representation as the V value, obtain the first generated image according to the third attention map and the second text embedding representation.

[0118] Continue to see Figure 8 After obtaining the third attention map, the third attention map can be used as a new attention map, and the second text embedding representation can be applied to the third attention map as the V value to obtain the output. For each input token, the iterative process from t=T to t=1 can be used as the first generated image.

[0119] 333, obtaining a first target loss according to the first generated image and the original image.

[0120] For example, see Figure 7 , the reconstruction loss between the first generated image 703 and the input image 701 can be used as the first target loss. Exemplarily, when the first LDM 702 is Figure 5 When the structure shown in , the first objective loss can use the formula (3) described above.

[0121] 334, updating the second text embedding representation according to the first target loss to obtain a third text embedding representation.

[0122] Specifically, the weight of the first LDM model can be fixed unchanged, and the second text embedding representation can be updated and optimized according to the first objective loss function until the stopping condition is met. For example, the V value corresponding to the second text embedding representation in step 3324 above can be updated. The updated second text embedding representation is the third text embedding representation. For example, continue to refer to Figure 7 , the embedding representation e of the source text description soc and the embedding representation e of the target text description tgt After being input into the denoising module in the first LDM702 together with the first random noise, the embedding representation e of the target text description can be obtained according to the reconstruction loss. tgt Update to obtain the optimized embedding representation e of the target text description opt .e opt This is the first example of the third text embedding representation. Since the third attention map retains the spatial information of the source map, the third text embedding representation updated according to the third attention map can contain the spatial information of the original image.

[0123] Optionally, the embodiment of the present application performs multiple iterative updates so that the third text embedding representation is a vector that is closest to the second text embedding representation and contains spatial information of the original image.

[0124] 340, input the original image and the third text embedding representation into the first text-generated image model, adjust the parameters of the first text-generated image model, and obtain the second text-generated image model.

[0125] Specifically, this step corresponds to the fine-tuning process of the text-to-image model. opt ) cannot restore the text-generated image model to the exact original image, but fine-tuning can make the model based on e opt The original image is restored exactly.

[0126] In some embodiments, see Fig.11 , the first text-to-image generation model can be fine-tuned according to the following steps 341 to 344 to obtain a second text-to-image generation model.

[0127] 341, adding noise to the original image and inputting it into a diffusion module to obtain a second random noise.

[0128] For example, see Fig.12 The original image may be the input image 701 in the figure, noise may be added to the input image 701 and then input into the first LDM 702, and the optimized text embedding representation e opt Input the first LDM702. Take the first LDM702 as Figure 5 As an example, the encoder E in the model first encodes the input image 701 (corresponding to x), maps the input image 701 from the pixel space to the latent space (corresponding to z), and then the diffusion module can diffuse the input image with added noise to obtain the first random noise corresponding to the input image (corresponding to z). T ).

[0129] 342, input the second random noise and the third text embedding representation into the attention layer in the denoising module, and generate a second generated image using the third text embedding representation as a text condition.

[0130] For example, the first LDM 702 is used as Figure 5 As an example, the optimized embedding representation of the target text description is opt The second random noise can be input into the denoising module in the first LDM702 together with the attention layer to achieve the effect of e opt As the text condition controls image generation, a second generated image 704 is obtained. Specifically, the fusion process of the attention layer on the text embedding representation and the random noise can be referred to the description above, which will not be repeated here.

[0131] 343, obtaining a second target loss according to the second generated image and the original image.

[0132] For example, see Fig.12, the reconstruction loss between the second generated image 704 and the input image 701 can be used as the second target loss. Exemplarily, when the first LDM 702 is Figure 5 When the structure shown in , the second objective loss can use the formula (3) described above.

[0133] 344 Update the parameters in the attention layer according to the second target loss to obtain a second text generation image model.

[0134] Specifically, the parameters of the attention layer in the first LDM model can be updated and optimized according to the second objective loss function until the stopping condition is met. Exemplarily, the parameter θ in the model can be updated. The updated first LDM is represented as the second LDM (an example of the second text generation image model). Optionally, the embodiment of the present application updates the second LDM multiple times to restore the original image with higher quality, that is, from e opt parse out more detailed information.

[0135] 350, obtaining a target image according to the second text embedding representation and the second text generating image model.

[0136] Specifically, the second text embedding representation can be input into the second text generation image model to obtain text conditional control of image editing according to the second text embedding representation to obtain a target image, thereby retaining key information of the original image while controlling the editing of the image for the target text.

[0137] In some embodiments, step 350 may be implemented as the following steps 351 and 352 .

[0138] 351, perform an interpolation operation on the second text embedding representation and the third text embedding representation to obtain a fourth text embedding representation.

[0139] From the above description, we can see that the third text embedding representation contains the spatial information of the original image, for example Figure 4 For the input image shown in FIG. 1 , the third text embedding representation can be the embedding representation vector that is closest to the text embedding representation vector of “abirdspreadingwings” and has the semantic information of the bird’s standing posture in the original image. At this time, the second text embedding representation corresponding to the target text description “abirdspreadingwings” has the semantic information of the bird’s wing-spreading posture. Therefore, by interpolating the two, the fourth text embedding representation obtained has semantic information between the bird’s wing-spreading posture and the standing posture.

[0140] For example, see Fig.13 , which can be used to embed the target text description tgt(an example of the second text embedding representation) and the optimized embedding representation e of the target text description opt (An example of the third text embedding representation) An interpolation operation is performed by the interpolation module 802. The result of the interpolation operation can be an example of the fourth text embedding representation.

[0141] 352, input the fourth text embedding representation and the third random noise into the second text generation image model to generate a target image, wherein the target image corresponds to the text description of the fourth text embedding representation.

[0142] For example, see Fig.13 , e tgt and e opt The result of the interpolation operation performed by the interpolation module 802 and the random noise 801 (an example of the third random noise) can be input into the second LDM 803 together. The second LDM 803 is an example of a second text generation image model. As an implementable method, the fourth text embedding representation can be used as a K value and a V value, and the third random noise can be input into the attention layer of the second LDM 803 as a Q value to obtain a target image 804.

[0143] At this time, the fourth text embedding representation is used as a text condition to control the editing of the image, and the target object described by the text corresponding to the fourth text embedding representation is obtained. As a specific example, for Figure 4 The input image and target description text in the present application embodiment can obtain an image of the bird in any state between (a) the standing posture of the bird in the image and (b) the spreading wings posture of the bird in the image.

[0144] In some embodiments, the second text embedding representation and random noise can be input into the second text generation image model to generate a target image. At this time, the second text embedding representation is used as a text condition to control the editing of the image, and the target image corresponding to the above target text description is obtained. As a specific example, for Figure 4 Based on the input image and target description text in (b), the embodiment of the present application can obtain an image of the bird in the wing-spreading posture in the image (b).

[0145] As an implementable method, the second text embedding representation can be used as the K value and the V value, and the third random noise can be input as the Q value into the attention layer of the second text generation image model (at this time, the attention map in the attention layer is the targetmap) to obtain the target image.

[0146] Fig.14A schematic diagram of the effect of an image obtained according to an embodiment of the present application is shown. Wherein, (a1) to (a6) are respectively input original images, and (b1) to (b6) are respectively target images obtained by editing (a1) to (a6) according to the target text description. Wherein, the target text description corresponding to the image (a1) is "a bird spreading wings", the target text description corresponding to the image (a2) is "a person giving the thumbs up", the target text description corresponding to the image (a3) ​​is "a goat jumping over a cat", the target text description corresponding to the image (a4) is "a sitting dog", the target text description corresponding to the image (a5) is "two kissing parrots", and the target text description corresponding to the image (a6) is "a children's drawing of a waterfall". Wherein, (a1) to (a5) are respectively expanded images, and (a6) is subjected to style conversion.

[0147] Therefore, the embodiment of the present application updates the second text embedding representation corresponding to the target text description according to the first text embedding representation of the source text description corresponding to the original image and the original image to obtain a third text embedding representation, so that the third text embedding representation can contain the spatial information of the original image, that is, the third text embedding representation can be used as a text condition to enable the model to generate the original image, but the original image generated by the model at this time is not accurate. Further, the parameters of the first text generation image model are adjusted according to the third text embedding representation to obtain a second text generation image model, and the second text generation image model can restore the original image according to the third text embedding representation. At this time, the original image generated by the model after fine-tuning is closer to the real original image (that is, a more accurate original image), that is, at this time, the model remembers the key information of the original image, so that according to the second text embedding representation and the second text generation model, the second text embedding representation can be used as a text condition to obtain a target image that retains the key information of the original image.

[0148] Therefore, the embodiments of the present application can fine-tune the pre-trained text-generated image model through the original image, the source text description corresponding to the original image, and the target text description, so that the model can control the editing of the picture according to the text to obtain the target image, and the target image can retain the key information of the original image, so that only text is needed without additional input conditions such as image masks or multiple input images, so as to control the editing of the original image, and retain the key information of the original image while controlling the editing of the image for the target text, thereby meeting the user's needs for image modification and lowering the user's usage threshold.

[0149] The specific implementation modes of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above implementation modes. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific implementation modes can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different implementation modes of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0150] It should also be understood that in the various method embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these sequence numbers can be interchanged where appropriate, so that the embodiments of the present application described can be implemented in an order other than those shown or described.

[0151] The above describes the method embodiment of the present application in detail. Figure 15 to Figure 16 , describe in detail the device embodiments of the present application.

[0152] Fig.15 1 is a schematic block diagram of a device 10 for generating an image from text according to an embodiment of the present application. Fig.15 As shown, the device 10 may include a first acquisition unit 11, an encoding unit 12, a pre-trained first text generation image model 13, an adjustment unit 14 and a second acquisition unit 15.

[0153] A first acquisition unit 11 is used to acquire an original image, a source text description corresponding to the original image, and a target text description; wherein the source text description is used to describe the screen content of the original image, and the target text description is used to describe the screen content of the target image generated by image editing the original image;

[0154] An encoding unit 12, configured to encode the source text description to obtain a first text embedding representation, and to encode the target text description to obtain a second text embedding representation;

[0155] A pre-trained first text-generated image model 13 is used to input the original image, the first text embedding representation and the second text embedding representation, the first text-generated image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and to update the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image;

[0156] An adjustment unit 14, configured to input the original image and the third text embedding representation into the first text-generated image model, and adjust parameters of the first text-generated image model to obtain a second text-generated image model;

[0157] The second acquisition unit 15 is used to generate an image model according to the second text embedding representation and the second text to obtain a target image.

[0158] In some embodiments, the first text generation image model 12 includes a diffusion module and a denoising module;

[0159] The first text-to-image model 12 is specifically used for:

[0160] Adding noise to the original image and inputting the image into the diffusion module to obtain a first random noise;

[0161] Inputting the first text embedding representation, the second text embedding representation and the first random noise into the denoising module, and obtaining a first generated image using the first text embedding representation and the second text embedding representation as text conditions;

[0162] Obtaining a first target loss according to the first generated image and the original image;

[0163] The second text embedding representation is updated according to the first target loss to obtain the third text embedding representation.

[0164] In some embodiments, the denoising module includes an attention layer;

[0165] The first text-to-image model 12 is specifically used for:

[0166] Input the first text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a first attention map, wherein the first attention map includes an attention map corresponding to each text input of the first text embedding representation;

[0167] Input the second text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a second attention map, wherein the second attention map includes an attention map corresponding to each text input of the second text embedding representation;

[0168] Obtaining a third attention map according to the first attention map and the second attention map;

[0169] The second text embedding representation is used as the V value, and the first generated image is obtained according to the third attention map and the second text embedding representation.

[0170] In some embodiments, the first text generation image model 12 is specifically used for:

[0171] If the description words in the source text description are replaced to obtain the target text description, the first attention map is determined as the third attention map.

[0172] In some embodiments, the first text generation image model 12 is specifically used for:

[0173] If the target text description is obtained by adding a description word to the source text description, the first attention map is inserted into the second attention map according to each text input position represented by the first text embedding to obtain the third attention map.

[0174] In some embodiments, the first text-to-image model includes a diffusion module and a denoising module; the denoising module includes an attention layer;

[0175] The adjustment unit 14 is specifically used for:

[0176] Adding noise to the original image and inputting the image into the diffusion module to obtain a second random noise;

[0177] Inputting the second random noise and the third text embedding representation into the attention layer in the denoising module, and generating a second generated image using the third text embedding representation as a text condition;

[0178] Obtaining a second target loss according to the second generated image and the original image;

[0179] The parameters in the attention layer are updated according to the second target loss to obtain the second text generation image model.

[0180] In some embodiments, the second acquiring unit 15 is specifically used for:

[0181] Performing an interpolation operation on the second text embedding representation and the third text embedding representation to obtain a fourth text embedding representation;

[0182] The fourth text embedding representation and third random noise are input into the second text generation image model to generate the target image, wherein the target image corresponds to the text description of the fourth text embedding representation.

[0183] In some embodiments, the third text embedding representation is a vector that is closest to the second text embedding representation.

[0184] In some embodiments, the first text-to-image model comprises a latent diffusion model.

[0185] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.15 The device 10 shown can execute the above method embodiment, and the above and other operations and / or functions of each module in the device 10 are respectively for implementing the corresponding processes in the above method 300, which will not be repeated here for the sake of brevity.

[0186] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0187] Fig.16 It is a schematic block diagram of an electronic device 30 provided in an embodiment of the present application.

[0188] like Fig.16 As shown, the electronic device 30 may include:

[0189] The memory 31 and the processor 32, the memory 31 is used to store the computer program and transmit the program code to the processor 32. In other words, the processor 32 can call and run the computer program from the memory 31 to implement the method in the embodiment of the present application.

[0190] For example, the processor 32 may be configured to execute the above method embodiments according to instructions in the computer program.

[0191] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0192] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0193] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0194] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0195] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0196] like Fig.16 As shown, the electronic device 30 may further include:

[0197] The transceiver 33 may be connected to the processor 32 or the memory 31 .

[0198] The processor 32 may control the transceiver 33 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 730 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more.

[0199] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0200] The present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0201] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0202] It is understandable that in the specific implementation of the present application, when the above embodiments of the present application are applied to specific products or technologies and involve relevant data such as user information, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards.

[0203] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0204] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0205] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0206] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating an image from text, characterized in that: include: Acquire an original image, a source text description corresponding to the original image, and a target text description; wherein the source text description is used to describe the screen content of the original image, and the target text description is used to describe the screen content of the target image generated by image editing the original image; Encoding the source text description to obtain a first text embedding representation, and encoding the target text description to obtain a second text embedding representation; Inputting the original image, the first text embedding representation and the second text embedding representation into a pre-trained first text generation image model, wherein the first text generation image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and updating the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image; Inputting the original image and the third text embedding representation into the first text-to-image generation model, adjusting the parameters of the first text-to-image generation model, and obtaining a second text-to-image generation model; An image model is generated according to the second text embedding representation and the second text to obtain the target image.

2. The method according to claim 1, characterized in that The first text generation image model includes a diffusion module and a denoising module; The step of inputting the original image, the first text embedding representation, and the second text embedding representation into a pre-trained first text generation image model to update the second text embedding representation to obtain a third text embedding representation includes: Adding noise to the original image and inputting the image into the diffusion module to obtain a first random noise; Inputting the first text embedding representation, the second text embedding representation and the first random noise into the denoising module, and obtaining a first generated image using the first text embedding representation and the second text embedding representation as text conditions; Obtaining a first target loss according to the first generated image and the original image; The second text embedding representation is updated according to the first target loss to obtain the third text embedding representation.

3. The method according to claim 2, characterized in that The denoising module includes an attention layer; The step of inputting the first text embedding representation, the second text embedding representation, and the first random noise into the denoising module, and obtaining a first generated image using the first text embedding representation and the second text embedding representation as text conditions includes: Input the first text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a first attention map, wherein the first attention map includes an attention map corresponding to each text input of the first text embedding representation; Input the second text embedding representation as the K value and the first random noise as the Q value into the attention layer to obtain a second attention map, wherein the second attention map includes an attention map corresponding to each text input of the second text embedding representation; Obtaining a third attention map according to the first attention map and the second attention map; The second text embedding representation is used as the V value, and the first generated image is obtained according to the third attention map and the second text embedding representation.

4. The method according to claim 3, characterized in that The step of obtaining a third attention map according to the first attention map and the second attention map comprises: If the description words in the source text description are replaced to obtain the target text description, the first attention map is determined as the third attention map.

5. The method according to claim 3, characterized in that: The step of obtaining a third attention map according to the first attention map and the second attention map comprises: If the target text description is obtained by adding a description word to the source text description, the first attention map is inserted into the second attention map according to each text input position represented by the first text embedding to obtain the third attention map.

6. The method according to claim 1, characterized in that The first text-to-image model includes a diffusion module and a denoising module; the denoising module includes an attention layer; The original image and the third text embedding representation are input into the first text generation image model, and the parameters of the first text generation image model are adjusted to obtain a second text generation image model, including: Adding noise to the original image and inputting the image into the diffusion module to obtain a second random noise; Inputting the second random noise and the third text embedding representation into the attention layer in the denoising module, and generating a second generated image using the third text embedding representation as a text condition; Obtaining a second target loss according to the second generated image and the original image; The parameters in the attention layer are updated according to the second target loss to obtain the second text generation image model.

7. The method according to claim 1, characterized in that The step of generating an image model according to the second text embedding representation and the second text to obtain a target image includes: Performing an interpolation operation on the second text embedding representation and the third text embedding representation to obtain a fourth text embedding representation; The fourth text embedding representation and third random noise are input into the second text generation image model to generate the target image, wherein the target image corresponds to the text description of the fourth text embedding representation.

8. The method according to claim 1, characterized in that The third text embedding representation is a vector that is closest to the second text embedding representation.

9. According to the method according to any one of claims 1-8, the first text generation image model includes a latent diffusion model.

10. A device for generating images from text, characterized in that: include: A first acquisition unit is used to acquire an original image, a source text description corresponding to the original image, and a target text description; wherein the source text description is used to describe the screen content of the original image, and the target text description is used to describe the screen content of the target image generated by image editing the original image; An encoding unit, configured to encode the source text description to obtain a first text embedding representation, and to encode the target text description to obtain a second text embedding representation; A pre-trained first text-generated image model is used to input the original image, the first text embedding representation and the second text embedding representation, the first text-generated image model is used to obtain a generated image with the first text embedding representation and the second text embedding representation as text conditions, and to update the second text embedding representation according to the generated image and the original image to obtain a third text embedding representation; wherein the third text embedding representation contains spatial information of the original image; an adjustment unit, configured to input the original image and the third text embedding representation into the first text-generated image model, and adjust parameters of the first text-generated image model to obtain a second text-generated image model; The second acquisition unit is used to generate an image model according to the second text embedding representation and the second text to obtain the target image.

11. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the processor executes the method according to any one of claims 1 to 9.

12. A computer storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for executing the method according to any one of claims 1 to 9.

13. A computer program product, characterized in that The method comprises a computer program code, and when the computer program code is executed by an electronic device, the electronic device executes the method according to any one of claims 1 to 9.

Images