Image generation method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses an image generation method and device, equipment and a storage medium, and is applied to the artificial intelligence technology, and the method comprises the steps: obtaining an image generation prompt text of a to-be-generated image; obtaining a style basic model obtained through training based on a style sample image associated with the first style information, a bypass module obtained through training based on a scene sample image associated with the scene information, and a target style model obtained through training based on a style sample image associated with the second style information; calling a bypass module and a style basic model, generating a prompt text according to the first random noise image and the image, and generating a pre-generated image matched with the scene information and the first style information; and calling the target style model, and generating a target image matched with the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image. According to the invention, the image generation quality can be improved.

Description

Technical Field

[0001] The present application relates to fields such as artificial intelligence technology, and in particular to an image generation method, device, equipment and storage medium. Background Art

[0002] Artificial intelligence software technology mainly includes computer vision technology, speech recognition technology, natural language processing technology, and machine learning / deep learning. Among them, computer vision technology usually includes image processing, image recognition, image generation, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, etc. Image generation refers to the diffusion model performing a series of operations on random noise images according to the description text to generate a predicted image related to the description text. However, in practice, it is found that due to the limited style sample images in a certain scene, the generalization ability of the diffusion model trained based on the style sample images is relatively weak, which in turn makes the image generation quality relatively low. Summary of the invention

[0003] The embodiments of the present application provide an image generation method, apparatus, device and storage medium to improve the quality of image generation.

[0004] An embodiment of the present application provides an image generation method, including:

[0005] Obtaining image generation prompt text of the image to be generated; the image generation prompt text reflects scene information, first style information and second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information;

[0006] Acquire a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information;

[0007] Calling the bypass module and the style basic model, generating a prompt text according to the first random noise image and the image, and generating a pre-generated image matching both the scene information and the first style information;

[0008] The target style model is called to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image.

[0009] An embodiment of the present application provides an image generation method, including:

[0010] Obtain N image-text pairs associated with the above scene information, an initial bypass module, and a basic generation model trained based on a general sample image; an image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1;

[0011] Calling the diffusion module of the above basic generation model to perform noise processing on the scene sample images in the above N image-text pairs to obtain N third noisy latent space representations;

[0012] Calling the denoising module of the basic generation model and the initial bypass module, and performing denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N predicted scene images;

[0013] According to the above-mentioned N predicted scene images and the sample scene images in the above-mentioned N image-text pairs, the parameters of the above-mentioned initial bypass module are adjusted to obtain a trained bypass module; the above-mentioned bypass module is used to implement the method described above.

[0014] An embodiment of the present application provides an image generating device, including:

[0015] A first acquisition module is used to acquire an image generation prompt text of the image to be generated; the image generation prompt text reflects the scene information, the first style information and the second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information;

[0016] A second acquisition module, configured to obtain a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information;

[0017] A first generating module is used to call the bypass module and the style basic model, generate a prompt text according to the first random noise image and the image, and generate a pre-generated image matching both the scene information and the first style information;

[0018] The second generation module is used to call the above-mentioned target style model to generate a target image matching the above-mentioned image generation prompt text according to the second random noise image, the above-mentioned image generation prompt text and the above-mentioned pre-generated image.

[0019] An embodiment of the present application provides an image generating device, including:

[0020] A first acquisition module is used to acquire N image-text pairs associated with the above scene information, an initial bypass module, and a basic generation model trained based on a general sample image; an image-text pair includes a scene sample image and a description text of the above scene sample image; N is a positive integer greater than 1;

[0021] A noise processing module is used to call the diffusion module of the basic generation model to perform noise processing on the scene sample images in the N image-text pairs to obtain N third noisy latent space representations;

[0022] A denoising processing module, used to call the denoising module of the above-mentioned basic generation model and the above-mentioned initial bypass module, and perform denoising processing on the above-mentioned N third noisy latent space representations according to the description texts in the above-mentioned N image-text pairs to obtain N predicted scene images;

[0023] An adjustment module is used to adjust the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module; the bypass module is used to implement the method described above.

[0024] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0025] On one hand, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0026] On the one hand, an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0027] In the present application, the style models (i.e., the style base model and the target style model) used for image style conversion and the bypass module used to generate the scene image are independently trained, i.e., the style base model is trained based on the style sample images associated with the first style information, the target style model is trained based on the style sample images associated with the second style information, and the bypass module is trained based on the scene sample images associated with the scene information. This can avoid the problem of too few sample images that meet both the scene information and the style information, resulting in low image generation quality, thereby improving the image generation quality. In practical applications, by embedding the bypass module into a style base model with better generalization performance to generate a pre-generated image that conforms to the semantic information, the semantic information refers to the scene information and the first style information, and through the target style model, the image style conversion is performed with the pre-generated image as the control information, and a target image that matches the image generation prompt text is obtained, so that the target image meets the scene requirements while meeting the image style requirements, thereby improving the image generation quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 is a schematic diagram of an image generation system provided by the present application;

[0030] Figure 2 It is a flowchart of an image generation method provided by the present application;

[0031] Figure 3 It is a flowchart of another image generation method provided by the present application;

[0032] Figure 4 This is a schematic diagram of an interaction scenario between a bypass unit and a first downsampling unit provided in the present application;

[0033] Figure 5 It is a flowchart of another image generation method provided by the present application;

[0034] Figure 6 It is a scene schematic diagram of an image generation method provided by the present application;

[0035] Figure 7 It is a flowchart of another image generation method provided by the present application;

[0036] Figure 8ais a schematic diagram of a scene sample image of multiple image styles provided by the present application;

[0037] Figure 8b It is a schematic diagram of normalizing the description text of a scene sample image provided by the present application;

[0038] Figure 8c It is a schematic diagram of the interaction between an initial bypass module and a basic generation model provided by the present application;

[0039] Fig. 9 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0040] Fig.10 is a structural schematic diagram of an image generating device provided in an embodiment of the present application;

[0041] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The embodiments of the present application may involve artificial intelligence technology, autonomous driving, smart transportation and other fields. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0044] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large image generation technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0045] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0046] This application mainly involves machine learning technology in artificial intelligence technology, which uses machine learning technology to generate images that meet both scene requirements and image style requirements.

[0047] It is understandable that in the present application, both the first style information and the second style information are used to indicate the image style of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information. The image style category indicated by the first style information can be called the first image style category, and the image style category indicated by the second style information can be called the second image style category. The first image style category reflects the rough image style of the image to be generated (the large category of image style), and the second image style category reflects the detailed image style of the image to be generated (the detailed category of image style). Images of different second image style categories have differences in details such as lines, colors, and shapes. For example, the first image style category is a cartoon style, and the second image style category is one or more of an old cartoon style (i.e., black and white), a modern cartoon style (color), a minimalist style, a picture book style, a comic style, etc. Alternatively, the first image style category can be a retro style, and the second image style category can include a graffiti style, a poster style, etc.; or the first image style category is a natural style, and the second image style can include a fallen leaf style, a rainy season style, a warm sun style, etc.

[0048] It is understandable that the style base model in this application can be called a style generation model, which is used to generate an image with a first image style category. The style base model is trained based on a style sample image associated with the first style information, that is, the style base model can be a generation model obtained by fine-tuning the diffusion model using a style sample image with a first image style category. The diffusion model (stable-diffusion) here can refer to a text-based image model based on a diffusion process, which can input a piece of text and generate a corresponding image through model reasoning. The diffusion model is the basic generation model in this application, also called a realistic basic model. The style base model can include stable-diffusion V1-5 (SD1-5), anythingV5, meinamix V9, etc. The target style model may also be referred to as a style transfer model, which is used to perform image style transfer on an image to obtain a target image having a second image style category. The target style model is trained based on a style sample image associated with the second style information, that is, the target style model may be a conversion model obtained by fine-tuning a diffusion model using a style sample image having a second image style category. The target style model may include stable-diffusion V1-5 (SD1-5), anythingV5, meinamix V9, etc.

[0049] It is understandable that the number of style sample images with the first image style category is much greater than that of style sample images with the second image style category, that is, the second image style category can also be called a rare image style, and the style sample images with the first image style category include style sample images with the second image style category. For example, if the first image style category is a cartoon style and the second image style is a minimalist style, the style sample images with the first style image category include cartoon images with a minimalist style, cartoon images with a picture book style, cartoon images with a modern cartoon style, etc.; the style sample images with the second image style category include cartoon images with a minimalist style. It can be seen that the style base model is obtained by training with style sample images of multiple sub-image style categories, and the target style model is obtained by training with style sample images of a single sub-image style category. Therefore, compared with the target style model, the style base model has better generalization performance; the target style model pays more attention to the details of a certain sub-image style category.

[0050] It is understandable that the bypass module in the present application refers to a module for generating scene images, and the bypass module can be trained based on scene sample images associated with scene information, and the scene information includes scene content information and scene atmosphere information, and the scene content information reflects the objects included in the scene sample image, and the objects can refer to one or more of buildings, people, plants (such as flowers, trees), animals (birds, cats, dogs, etc.), natural scenery (such as the sea, mountains, waterfalls), transportation tools (such as cruise ships, cars, airplanes), etc. The scene atmosphere information is used to reflect the scene parameters corresponding to the scene where the object in the scene sample image is located, such as scene parameters including lighting, time (such as sunset, sunrise), weather, etc.

[0051] In practice, it is found that due to the limited sample images of style in a certain scene, that is, there are relatively few sample images with both a specific scene and a specific style, the generalization ability of the diffusion model trained based on the style sample images is relatively weak, and thus, the quality of image generation is relatively low. For example, when it is necessary to train a style generation model that generates cartoon images with cruise ships, since there are relatively few cartoon sample images with cruise ships, the accuracy of the style generation model is relatively low, and thus the generated image does not meet the requirements. Based on this, the present application proposes an image generation method, which has the following advantages: 1. Independently train the style model (i.e., the style base model and the target style model) used for image style conversion, and the bypass module for generating scene images, so as to avoid the problem of too few sample images that meet both scene information (scene requirements) and style information (style requirements), resulting in relatively low image generation quality, thereby improving the image generation quality. 2. By embedding the bypass module into a style base model with better generalization performance, a pre-generated image that conforms to the semantic information is generated. Through the target style model, the pre-generated image is used as the control information to perform image style conversion, and a target image that matches the image generation prompt text is obtained, so that the target image meets the scene requirements and the image style requirements, thereby improving the image generation quality. 3. The pre-generated image meets the scene requirements and the rough image style requirements (i.e., the first image style category), and the target image meets the scene requirements, the rough image style requirements, and the detailed image style requirements (i.e., the second image style category). When the pre-generated image meets the scene requirements and the rough image style, it gradually approaches the final image style requirements, thereby improving the quality of the generated image.

[0052] In order to facilitate a clearer understanding of the present application, the image generation system implementing the present application is first introduced. Figure 1 As shown, the image generation system includes a server 10 and a terminal cluster, and the terminal cluster may include one or more terminals, and the number of terminals is not limited here. Figure 1As shown, the terminal cluster may specifically include terminal 1, terminal 2, ..., terminal n; it can be understood that terminal 1, terminal 2, terminal 3, ..., terminal n can all be connected to the server 10 through a network connection, so that each terminal can exchange data with the server 10 through the network connection.

[0053] One or more target applications are installed in the terminal. The target application here may refer to an application with image processing (such as downloading images, generating images) functions. For example, the target application may include an independent application, a web application, a small program in a host application, etc. The server 10 refers to a device that provides backend services for the target application in the terminal. In one embodiment, the server 10 may train the initial bypass module according to the scene sample image and the basic generation model to obtain the bypass module. The server 10 may send the bypass module to the terminal, and the terminal may generate an image based on the bypass module.

[0054] Optionally, the image generation method in the present application includes a training process for the initial bypass module and a target image generation process, and the training process and the generation process can be performed by the server 10, or by the terminal, or by the terminal and the server 10 together. Figure 2 As shown, the training process is performed by the server 10 and the generation process is performed by the terminal as an example. Figure 2 As shown, the training process includes the following steps s11 to s13; the generation process includes the following steps s14 and s15:

[0055] s11. Obtain image-text pairs; that is, the server 10 can obtain N image-text pairs associated with the scene information, and one image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1. s12. Normalize the description text; the server 10 separately extracts the scene content information and the scene atmosphere information from the description text in the image-text pair to realize the normalized description text. The specific implementation process can refer to the following step S201. s13. Fine-tune the initial bypass module based on the basic generation model. The server 10 can fine-tune the initial bypass module based on N image-text pairs and the basic generation module to obtain the bypass module; the specific implementation process can refer to the following steps S202 to S204.

[0056] Further, s14, pre-generate under the bypass module using the style base model. After the server 10 obtains the bypass module, the bypass module can be sent to any terminal. The terminal can generate a pre-generated image that meets the semantic information of the image generation prompt text based on the image generation prompt text through the style base model and the bypass module. The specific implementation process can refer to the following step S103. s15, generate the target image under the edge map control module using the target style model; since the scene information in the image generation prompt text is more similar to the scene information of the scene sample image used to train the initial bypass module, the pre-generated image meets the scene requirements and also meets the image generation prompt text (that is, it meets the first image style category), but the image style does not meet the rare style requirements of the target generation requirements (that is, the second image style category). Therefore, the terminal is also required to generate the target image based on the image generation prompt text under the control of the pre-generated image through the target style model and the edge map control module. The specific implementation process can refer to the following step S104.

[0057] It is understandable that the server can be an independent physical server, or a server cluster or distributed system composed of at least two physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Network (CDN), and big data and artificial intelligence platforms and other basic cloud computing services. The terminal can specifically refer to a vehicle-mounted terminal, a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a screen speaker, a smart TV, a smart watch, etc., but is not limited to this. Each terminal and server can be directly or indirectly connected by wired or wireless communication. At the same time, the number of terminals and servers can be one or at least two, and this application is not limited here.

[0058] For further information, see Figure 3 , is a flow chart of an image generation method provided in an embodiment of the present application. Figure 3 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 can be executed by the server in Figure 1 The terminal and server in the terminal cluster are executed in collaboration, and the devices used to execute the image generation method in this application can be collectively referred to as computer devices. The method may include the following steps:

[0059] S101, obtaining image generation prompt text of an image to be generated; the image generation prompt text reflects scene information, first style information, and second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information.

[0060] In the present application, when it is necessary to generate an image of a specific scene or a specific style, the computer device can obtain the image generation prompt text of the image to be generated. The image generation prompt text can be input by the user in the target application mentioned above. The image generation prompt text is used to reflect the scene information, the first style information, and the second style information of the image to be generated. The text type of the image generation prompt text can be Chinese or English. This application takes the image generation prompt text in Chinese as an example for explanation. For example, the image generation prompt text is: sunrise on the sea level, and the image style is the warm sun style under the natural style, that is, the "sunrise on the sea level" in the image generation prompt text is the scene information, the "natural style" is the first style information, and the "warm sun style" is the second style information. For the description of the scene information, the first style information, and the second style information, please refer to the above description, and the repeated parts will not be repeated.

[0061] S102: Acquire a style base model trained based on style sample images associated with the first style information, a bypass module trained based on scene sample images associated with the scene information, and a target style model trained based on style sample images associated with the second style information.

[0062] In the present application, a computer device can obtain, from a pre-trained model library, a style base model trained based on a style sample image associated with the first style information, a bypass module trained based on a scene sample image associated with the scene information, and a target style model trained based on a style sample image associated with the second style information. That is, the style base model has the ability to generate an image with a first image style category, the target style model has the ability to convert the image style of the image into a second image style category, and the bypass module has the ability to generate an image associated with the scene information. The style base model, the bypass module, and the target style model are independently trained, which can avoid the problem of too few sample images that simultaneously meet the scene information (scene requirements) and the style information (style requirements), resulting in relatively low image generation quality, thereby improving the image generation quality.

[0063] It should be noted that the image style category also includes the real image style category. The real image style category can refer to the image style of the image taken in the original camera mode, that is, the real image style category refers to the style of the image that has not been stylized; usually, images in the real image style category are more common, and images in non-real image styles (such as cartoon style and retro style) are relatively rare. Therefore, the real image style category can be called a common style, and the non-real style can be called a rare style.

[0064] Optionally, the above-mentioned acquisition of the style basic model obtained by training based on the style sample image associated with the first style information includes: the computer device can determine the style difference between the image style type indicated by the first style information and the real image style category; the computer device can obtain the difference between the image with the first image style category and the image with the real image style category, and determine the style difference between the image style type indicated by the first style information and the real image style category according to the difference. If the style difference is greater than the difference threshold, indicating that the first image style category is more inclined to a rare style, the style generation model obtained by training based on the style sample image associated with the first style information is determined as the style basic model; that is, one rare style corresponds to one style generation model, which is conducive to making the image style category of the pre-generated image closer to the first image style category and improving the image generation quality. If the style difference is less than or equal to the difference threshold, indicating that the first image style category is more inclined to the common style (real image style category), the basic generation model trained based on the universal sample image is determined as the style basic model; the universal sample image includes the style sample image associated with the above-mentioned first style information. Since the basic generation model has a stronger generalization ability, this is conducive to improving the quality of the pre-generated image.

[0065] S103: calling the bypass module and the style base model, generating a pre-generated image matching both the scene information and the first style information according to the first random noise image and the image generation prompt text.

[0066] In the present application, since there are relatively few style sample images with the second image style category, the generalization ability of the target style model is relatively poor. Therefore, if the bypass module and the target style model are directly used to generate a target image that matches the image generation prompt information, it will lead to insufficient stylization, that is, it is impossible to generate an image with the target style (i.e., the second image style category). Based on this, the computer device can call the bypass module and the style base model to generate a pre-generated image that matches the first style information and the scene information according to the first random noise image and the image generation prompt text. The pre-generated image meets the scene requirements and the first image style category requirements. It can also be said that the pre-generated image meets the semantic information of the image generation prompt text, that is, the pre-generated image does not meet the second image style category requirements. By using the bypass module with the help of a style base model with better generalization performance, a pre-generated image that meets the semantic information is generated, which is conducive to the subsequent accurate generation of a target image that meets the second image style category requirements and the scene requirements, and improves the image generation quality.

[0067] In one embodiment, the above-mentioned calling of the bypass module and the style base model, generating a prompt text according to the first random noise image and the image, and generating a pre-generated image matching both the scene information and the first style information, includes: the computer device can generate a random number, generate a first random noise image based on the random number, encode the first random noise image, and obtain a first noisy latent space representation of the first random noise image, and the first noisy latent space representation reflects the vector of color information and texture information of the first random noise image. Further, the computer device can call the bypass module and the denoising module of the style base model, generate the prompt text according to the image, and perform T denoising on the first noisy latent space representation to obtain a first denoised latent space representation; T is a positive integer. The denoising module can be a module for denoising the noisy latent space representation, and the first denoised latent space representation reflects the vector of color information and texture information of the pre-generated image. Then, the computer device can decode the first denoised latent space representation to obtain a pre-generated image matching both the scene information and the first style information.

[0068] Optionally, the denoising module of the style base model includes a first upsampling network and a first downsampling network, and the network structure of the bypass module can be the same as the network structure of the first downsampling network, that is, the bypass module is also a downsampling network, the downsampling network is used to extract global features associated with the image generation prompt text, and the upsampling network is used to extract local features associated with the image generation prompt text, the local features include contour information, and the local features include texture information and color information.

[0069] Optionally, the above-mentioned calling of the bypass module and the denoising module of the style base model, generating a prompt text according to the image, and performing T denoising processes on the first noisy latent space representation to obtain a first denoised latent space representation includes: for the jth denoising process, the computer device may generate a prompt text and an input representation P from the image j Input to the bypass module to obtain the output data of the bypass module in the jth denoising process; the first input representation is the first noisy latent space representation, when j is not 1, the input representation P j is the output data of the first upsampling network after the j-1th denoising process, where j is a positive integer less than or equal to T. The image generates a prompt text and the input representation P j Input the output data of the first down-sampling network in the jth denoising process to the first down-sampling network, and obtain the output data of the first down-sampling network in the jth denoising process. Input the output data of the bypass module in the jth denoising process, the output data of the first down-sampling network in the jth denoising process, and the image generation prompt text to the first up-sampling network, and obtain the output data of the first up-sampling network in the jth denoising process; repeat the above steps until the output data of the first up-sampling network in the Tth denoising process is obtained, and determine the output data of the first up-sampling network in the Tth denoising process as the first denoising latent space representation.

[0070] It should be noted that since the bypass module is trained based on scene sample images associated with scene information, and the style base model is trained based on style sample images associated with the first style information, that is, the bypass module and the first downsampling network in the style base module are trained based on different sample images, the parameters of the bypass module and the parameters of the first downsampling network in the style base module are different. Therefore, the output data of the bypass module in the jth denoising process is different from the output data of the first downsampling network in the jth denoising process. The output data of the bypass module is a global feature related to the scene information, and the output data of the first downsampling network is a global feature associated with the first image style category. As j increases, the global features included in the output data of the bypass module and the first downsampling network increase accordingly, and the global features and local features included in the output data of the first upsampling network also increase accordingly.

[0071] For example, for the first denoising process, the computer device may input the image generation prompt text and the first noisy latent space representation (i.e., the first input representation) into the bypass module, and the bypass module downsamples the first latent space representation according to the image generation prompt text to obtain the output data of the bypass module in the first denoising process, that is, the output data of the bypass module in the first denoising process includes the global features associated with the image generation prompt text. The image generation prompt text and the first noisy latent space representation are input into the first downsampling network, and the first downsampling network downsamples the first noisy latent space representation according to the image generation prompt text to obtain the output data of the first downsampling network in the first denoising process, and the output data of the first downsampling network in the first denoising process includes the global features associated with the image generation prompt text. Further, the output data of the bypass module in the first denoising process, the output data of the first downsampling network in the first denoising process and the image generation prompt text are input into the first upsampling network, the first upsampling network fuses the output data of the bypass module in the first denoising process and the output data of the first downsampling network in the first denoising process to obtain the first fused data 1, the first upsampling network generates prompt text according to the image, and upsamples the first fused data 1 to obtain the output data of the first upsampling network in the first denoising process, the output data of the first upsampling network in the first denoising process includes global features and local features associated with the image generation prompt text, and the first denoising process is terminated at this point.

[0072] Further, for the second denoising process, the output data of the first upsampling network in the first denoising process is used as the second input representation, and the computer device can input the image generation prompt text and the second input representation into the bypass module, and the bypass module downsamples the second input representation according to the image generation prompt text to obtain the output data of the bypass module in the second denoising process. The image generation prompt text and the second input representation are input into the first downsampling network, and the first downsampling network downsamples the second input representation according to the image generation prompt text to obtain the output data of the first downsampling network in the second denoising process. Further, the output data of the bypass module in the second denoising process, the output data of the first downsampling network in the second denoising process and the image generation prompt text are input into the first upsampling network, the first upsampling network fuses the output data of the bypass module in the second denoising process and the output data of the first downsampling network in the second denoising process to obtain the first fused data 2, the first upsampling network performs upsampling processing on the first fused data 2 according to the image generation prompt text, and obtains the output data of the first upsampling network in the second denoising process, the output data of the first upsampling network in the second denoising process includes global features and local features associated with the image generation prompt text, and the second denoising process ends here. Repeat the above steps until the output data of the first upsampling network in the Tth denoising process is obtained, and the output data of the first upsampling network in the Tth denoising process is determined as the first denoising latent space representation.

[0073] Optionally, the bypass module includes S bypass units, and the first downsampling network includes S first downsampling units, where S is a positive integer; one bypass unit corresponds to one first downsampling unit, and the input data of the bypass unit and the first downsampling unit with a corresponding relationship are the same, and the output data of the bypass unit will be added to the output data of the corresponding first downsampling unit. For example, the ath bypass unit corresponds to the first downsampling unit a, and in the jth denoising process, when a=1, the input data of the first bypass unit and the first downsampling unit 1 are both the image generation prompt text and the input representation P j When a≠1, the input data of the ath bypass unit and the first downsampling unit a include the image generation prompt text, the output data of the a-1th bypass unit in the jth denoising process, and the output data of the first downsampling unit a in the jth denoising process. Input representation P j It refers to the output data of the first upsampling network after the j-1th denoising process.

[0074] Optionally, the image generates prompt text and input representation P jInput to the first down-sampling network to obtain the output data of the first down-sampling network after the j-th denoising process, including: Figure 4 As shown, the computer device can input the output data of the ath bypass unit in the bypass module in the jth denoising process, the output data of the first downsampling network a in the jth denoising process, and the image generation prompt text into the first downsampling network a+1 to obtain the output data of the first downsampling network a+1 in the jth denoising process; a is a positive integer less than S. That is, the first downsampling network a+1 can fuse the output data of the ath bypass unit in the bypass module in the jth denoising process and the output data of the first downsampling network a in the jth denoising process to obtain the second fused data a, generate the prompt text according to the image, downsample the second fused data a, and obtain the output data of the first downsampling network a+1 in the jth denoising process. Further, repeat the above steps until the output data of the first downsampling network S in the jth denoising process is obtained, and the output data of the first downsampling network S in the jth denoising process is determined as the output data of the first downsampling network in the jth denoising process. The input representation P j The prompt text generated by the above image is the input data of the first downsampling network 1 in the above j-th denoising process. The first downsampling network S is a downsampling network among the S first downsampling networks connected to the first upsampling network, and can also be called the last downsampling network.

[0075] S104 , calling a target style model, and generating a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text, and the pre-generated image.

[0076] In the present application, after obtaining a pre-generated image that meets the semantic information, the computer device can call the target style model, take the pre-generated image as the control information, and generate a target image that matches the image generation prompt text according to the second random noise image and the image generation prompt text, that is, the target image meets the scene requirements and the second image style category requirements. Since the second image style category is a subcategory of the first image style category, the target image also meets the first image style category requirements. When the pre-generated image meets the scene requirements and the rough image style, it gradually approaches the final image style requirements, which can improve the quality of the generated image.

[0077] It should be noted that the first random noise image and the second random noise image both refer to a randomly generated noise image. The random noise image can be generated by a random number. Different random numbers correspond to different random noise images. The random number refers to an arbitrary number. Random noise images corresponding to different random numbers have different image features. The image features may refer to image style, image color, etc.

[0078] In one embodiment, the above-mentioned calling the above-mentioned target style model, generating a target image matching the above-mentioned image generation prompt text according to the second random noise image, the above-mentioned image generation prompt text and the above-mentioned pre-generated image, includes: for example, Figure 5 In the embodiment, after the computer device generates a pre-generated image based on the image generation prompt text through the style base model and the bypass module, it can generate a prompt text based on the image and generate a target image through the target style model and the edge map control module, taking the pre-generated image as control information. The edge map control module (controlNet) refers to a module for edge extraction. The network structure of the edge map control module can be the same as the network structure of the downsampling network of the target style model. The edge map control module can be a canny2img module. Specifically, the computer device can input the pre-generated image into the edge map control module, and the edge map control module performs L edge extractions on the pre-generated image to obtain L edge images reflecting the contour information in the pre-generated image; L is a positive integer, and the edge image is used to reflect the contour information of the pre-generated image. The contour information matches the image generation prompt text. For example, if the image generation prompt text includes a sun road on the sea level, then the contour information includes a contour reflecting the sea level and a contour of the sunset.

[0079] Furthermore, the computer device can generate a random number, which can be any value, obtain a random noise image corresponding to the random number as a second random noise image, encode the second random noise image, and obtain a second noisy latent space representation corresponding to the second random noise image. Then, the denoising module of the target style model can be called, and the second noisy latent space representation is denoised L times according to the L edge images and the prompt text generated by the image to obtain a second denoised latent space representation that matches the prompt text generated by the image; that is, the denoising module of the target style model uses the L edge images and the prompt text generated by the image as control information, performs L denoising on the second noisy latent space representation, obtains a second denoised latent space representation that matches the prompt text generated by the image, and decodes the second denoised latent space representation to obtain a target image that matches the prompt text generated by the image. By generating prompt information from L edge images and image text to generate a target image, it is possible to avoid the loss of semantic information in the pre-generated image when the target style model is performing style conversion, causing the target image to fail to meet the scene requirements, thereby improving the image generation quality. In addition, the final image style requirements can be gradually approached, which can improve the quality of the generated image.

[0080] Optionally, the above-mentioned calling of the denoising module of the above-mentioned target style model, performing L denoising processes on the above-mentioned second noisy latent space representation according to the above-mentioned L edge images and the above-mentioned image generation prompt text, and obtaining a second denoised latent space representation that matches the above-mentioned image generation prompt text, includes: the computer device can obtain the denoised latent space representation i obtained by the target style model in the i-th denoising process for the i+1-th denoising process; the first denoised latent space representation is obtained according to the above-mentioned image generation prompt text, the second noisy latent space representation and the first edge image, i is a positive integer less than L; further, calling the denoising module of the target style model, performing denoising process on the denoised latent space representation i according to the i-th edge image and the above-mentioned image generation prompt text, and obtaining the denoised latent space representation i+1. Repeat the above steps until the denoised latent space representation obtained by the L-th denoising process is obtained, and the denoised latent space representation obtained by the L-th denoising process is determined as the second denoised latent space representation that matches the image generation prompt text.

[0081] For example, Figure 6 As shown, the image-generated prompt text is: a cruise ship on the sea level at sunset, cartoon style, that is, the scene information is: sunset, sea level, cruise ship; the first style information is cartoon style, and the second style information is cartoon style. The computer device can generate a pre-generated image 62a based on the first random noise image 61a and the image-generated prompt text through the basic style model and the bypass module. The pre-generated image 62a satisfies the semantic information of the image generation prompt information, that is, the pre-generated image 62a includes a cruise ship located on the sea level at sunset, and at the same time, has a cartoon style. Further, the pre-generated image 62a can be input into the edge map control module, and the edge map control module extracts L edge images from the pre-generated image 62a. Figure 6 In the description, an edge image is taken as an example, namely edge image 63a, which includes the contour information of the object in the pre-generated image 62a. Further, the second random noise image 64a, the image generation prompt text and the edge image 63a are input into the target style model. The target style model uses the edge image 63a as control information and generates the target image 65a according to the second random noise image 64a and the image generation prompt text. Figure 6 As shown, compared with the pre-generated image 62a, the target image 65a is more three-dimensional, and the image style of the target image 65a is closer to the cartoon style.

[0082] In the present application, the style models (i.e., the style base model and the target style model) used for image style conversion and the bypass module used to generate the scene image are independently trained, i.e., the style base model is trained based on the style sample images associated with the first style information, the target style model is trained based on the style sample images associated with the second style information, and the bypass module is trained based on the scene sample images associated with the scene information. This can avoid the problem of too few sample images that meet both the scene information and the style information, resulting in low image generation quality, thereby improving the image generation quality. In practical applications, by embedding the bypass module into a style base model with better generalization performance to generate a pre-generated image that conforms to the semantic information, the semantic information refers to the scene information and the first style information, and through the target style model, the image style conversion is performed with the pre-generated image as the control information, and a target image that matches the image generation prompt text is obtained, so that the target image meets the scene requirements while meeting the image style requirements, thereby improving the image generation quality.

[0083] For further information, see Figure 7 , is a flow chart of an image generation method provided in an embodiment of the present application. Figure 7 As shown, this method can be Figure 1 It can be executed by any terminal in the terminal cluster in Figure 1 can be executed by the server in Figure 1 The terminal and server in the terminal cluster are executed in collaboration, and the devices used to execute the image generation method in this application can be collectively referred to as computer devices. The method may include the following steps:

[0084] S201, obtaining N image-text pairs associated with the scene information, an initial bypass module, and a basic generation model trained based on a general sample image; an image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1.

[0085] In the present application, the N scene sample images in the N image-text pairs belong to a variety of image style categories, such as cartoon style, natural style, retro style, etc. The description text of the scene sample image is used to describe the scene content and scene atmosphere of the scene sample image, etc. The network structure of the initial bypass module can be the same as the network structure of the downsampling network in the denoising module of the basic generation model, and the initial parameters of the initial bypass module can be the same as the parameters of the downsampling network in the denoising module of the basic generation model. The computer device can obtain N image-text pairs associated with the scene information, the initial bypass module, and the basic generation model trained based on the general sample image from a web page or local storage.

[0086] For example, Figure 8aAs shown, when the scene information is: a cruise ship at sunset, the scene sample images may include scene sample image 81a, scene sample image 82a, and scene sample image 83a, all of which include a cruise ship, and the image styles of scene sample image 81a, scene sample image 82a, and scene sample image 83a are comic style, nature style, and picture book style, respectively.

[0087] Optionally, the above-mentioned acquisition of N image-text pairs associated with the above-mentioned scene information includes: the computer device can acquire N scene sample images associated with the scene information, and the initial description texts corresponding to the N scene sample images. Since the N scene sample images belong to a variety of different image style categories, in order to avoid the overfitting of the bypass module obtained by training due to the scene atmosphere information affecting the learning ability of the target scene. Based on this, the computer device can perform normalization processing on the N initial description texts, that is, the scene atmosphere information in the N initial description texts is extracted separately, that is, the scene content information and scene atmosphere information corresponding to the scene sample image c can be extracted from the initial description text corresponding to the scene sample image c, so that the overfitting problem of the training process can be avoided, and c is a positive integer less than or equal to N. Further, the scene content information and scene atmosphere information corresponding to the scene sample image c are determined as the description text of the scene sample image c. According to the N scene sample images and the description texts of the N scene sample images, N image-text pairs are generated, that is, the scene sample image c and the description text of the scene sample image c constitute a image-text pair.

[0088] For example, Figure 8b In the example, the initial description text of the scene sample image 84a is: cruise ship at sunset. If the scene atmosphere information is not separated separately, the initial bypass module and the basic generation module will think that "ship at dusk" is a cruise ship, resulting in a relatively low quality of image generation. Figure 8b As shown, the initial description text of the scene sample image 84a: cruise ship at sunset, can be normalized to: sunset, cruise ship.

[0089] S202, calling the diffusion module of the basic generation model to perform noise processing on the scene sample images in the N image-text pairs to obtain N third noisy latent space representations.

[0090] In the present application, the computer device can fuse the scene sample images in the N image-text pairs with the random noise images to obtain N noisy sample images, encode the N noisy sample images to obtain N initial latent space representations, perform T noise addition processing on the N initial latent space representations to obtain N third noisy latent space representations, and the N third noisy latent space representations mainly include noise data subjected to T noise addition processing.

[0091] S203, calling the denoising module of the basic generation model and the initial bypass module, and performing denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images.

[0092] In the present application, a computer device can input N noisy latent space representations and N description texts into a denoising module and an initial bypass module of a basic generation model. The denoising module of the basic generation model and the initial bypass module perform denoising processing on the N third noisy latent space representations according to the N description texts to obtain N predicted scene images, where the N predicted scene images refer to images that match the corresponding description texts.

[0093] Optionally, the above-mentioned calling of the denoising module of the above-mentioned basic generation model and the initial bypass module, and performing denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images, includes: the computer device can call the denoising module of the above-mentioned basic generation model and the initial bypass module, and perform T times denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N third denoised latent space representations; the third denoised latent space representation refers to a vector reflecting the image features of the predicted scene image, and the N third denoised latent space representations are decoded to obtain N predicted scene images.

[0094] Optionally, the denoising module of the basic generative model includes a second upsampling network and a second downsampling network; the network structure of the bypass module can be the same as the network structure of the second downsampling network. The above-mentioned calling of the denoising module of the above-mentioned basic generative model and the above-mentioned initial bypass module performs T denoising processes on the above-mentioned N third noisy latent space representations according to the description texts in the above-mentioned N image-text pairs to obtain N third denoised latent space representations, including: for the jth denoising process, the description text k in the kth image-text pair and the input representation Q j Input to the initial bypass module to obtain the output data of the initial bypass module in the jth denoising process; the first input representation is the third noisy latent space representation k, j is a positive integer less than or equal to T. That is, the initial bypass module, according to the description text k, j Downsampling is performed to obtain the output data of the initial bypass module in the jth denoising process. j Input to the second downsampling network to obtain the output data of the second downsampling network in the above j-th denoising process; that is, the second downsampling network, according to the description text k, represents the input Q jPerform downsampling processing to obtain the output data of the initial bypass module in the jth denoising processing, where k is a positive integer less than or equal to N. Then, the output data of the initial bypass module in the jth denoising processing, the output data of the second downsampling network in the jth denoising processing, and the description text k are input into the second upsampling network to obtain the output data of the second upsampling network in the jth denoising processing; the second downsampling network fuses the output data of the initial bypass module in the jth denoising processing and the output data of the second downsampling network in the jth denoising processing to obtain the third fused data, and the third fused data is upsampled according to the description text k to obtain the output data of the second upsampling network in the jth denoising processing. Repeat the above steps until the output data of the second upsampling network in the jth denoising processing is obtained, and the output data of the second upsampling network in the Tth denoising processing is determined as the third denoising latent space representation associated with the kth image-text pair.

[0095] Optionally, the initial bypass module includes S bypass units, and the second downsampling network includes S second downsampling units, where S is a positive integer; one bypass unit corresponds to one second downsampling unit, and the input data of the bypass unit and the second downsampling unit with a corresponding relationship are the same, and the output data of the bypass unit will be added to the output data of the corresponding second downsampling unit. For example, the ath bypass unit corresponds to the second downsampling unit a, and in the jth denoising process, when a=1, the input data of the first bypass unit and the second downsampling unit 1 are both the description text k and the input representation Q j When a≠1, the input data of the ath bypass unit and the second downsampling unit a include the description text k, the output data of the a-1th bypass unit at the jth denoising process, and the output data of the second downsampling unit a at the jth denoising process; the input representation Q j It refers to the output data of the second upsampling network after the j-1th denoising process.

[0096] The above description text k and input representation Q jInputting into the second downsampling network to obtain the output data of the second downsampling network in the j-th denoising process includes: the computer device can input the output data of the a-th bypass unit in the initial bypass module in the j-th denoising process, the output data of the second downsampling unit a in the j-th denoising process, and the description text k into the second downsampling unit a+1 to obtain the output data of the second downsampling unit a+1 in the j-th denoising process; a is a positive integer less than S. That is, the second downsampling network a+1 can fuse the output data of the a-th bypass unit in the initial bypass module in the j-th denoising process and the output data of the second downsampling network a in the j-th denoising process to obtain the third fused data a, and downsample the third fused data a according to the description text k to obtain the output data of the second downsampling network a+1 in the j-th denoising process. Further, the above steps are repeated until the output data of the second down-sampling network S in the j-th denoising process is obtained, and the output data of the second down-sampling network S in the j-th denoising process is determined as the output data of the second down-sampling network in the j-th denoising process. j And the description text k is the input data of the second downsampling unit 1 in the j-th denoising process, and the second downsampling network S is a downsampling network among the S second downsampling networks connected to the second upsampling network, and can also be called the last downsampling network.

[0097] S204. According to the N predicted scene images and the sample scene images in the N image-text pairs, the parameters of the initial bypass module are adjusted to obtain a trained bypass module; the bypass module is used to implement the above method.

[0098] In the present application, the computer device can adjust the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the above-mentioned N image-text pairs to obtain the trained bypass module, that is, only the parameters of the initial bypass module are adjusted, which is conducive to preserving the scene-related knowledge in the trained bypass module, and only needs to fine-tune the initial bypass model once. The bypass module can be embedded in different style models to generate images that meet the scene requirements and style requirements, improve the utilization rate of the bypass module, and do not need to undergo multiple fine-tunings, which is more convenient to use and can save resources. In other words, the bypass module obtained after an offline fine-tuning can adapt to the needs of image style conversion of various downstream style models, and does not need to fine-tune a bypass module for each image style category, which is convenient for application. For example, the target style model is obtained by training a cartoon sample image based on a minimalist style. When a cartoon image in a picture book style needs to be generated, it is only necessary to replace the above target style model with a style model obtained by training a cartoon sample image based on a picture book style. This is conducive to generating images that meet the scene requirements and any style requirements, and is more convenient to apply.

[0099] In one embodiment, the above-mentioned adjustment of the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain the trained bypass module includes: the computer device can determine the loss function value of the initial bypass module according to the difference between the N predicted scene images and the sample scene images in the N image-text pairs; the loss function value is used to measure the accuracy of the initial bypass module in generating the scene image, that is, the larger the loss function value, the lower the accuracy of the initial bypass module in generating the scene image; the smaller the loss function value, the higher the accuracy of the initial bypass module in generating the scene image. Therefore, the computer device can perform multiple rounds of iterative adjustments on the parameters of the initial bypass module according to the loss function value to obtain the trained bypass module, which can improve the accuracy of the bypass module in generating the scene image.

[0100] It should be noted that multiple rounds of iterative adjustment refer to performing E rounds of iterative adjustments on the initial bypass module for N images and texts. The training of N image-text pairs in the initial bypass module is called one round of iterative adjustment. Due to the limited video memory resources of computer equipment, it is impossible to train all data (N image-text pairs) at the same time. Therefore, it is necessary to train the full amount of data (i.e., N image-text pairs) in batches, such as m training batches. Each training batch corresponds to multiple image-text pairs. The parameters of the initial bypass module are updated once for each training batch until all image-text pairs have completed one round of iterative adjustment. Figure 8c As shown, the specific training process includes the following steps:

[0101] (1) Parameter initialization is performed before the first training batch of the first round of iterative adjustment: the image encoder, image decoder, text encoder, and basic generative model are all trained, and their parameters do not need to be initialized or adjusted in the subsequent training process. The parameters of the initial bypass module can be initialized based on the parameters of the downsampling network in the denoising module of the basic generative model, that is, the parameters of the downsampling network in the denoising module of the basic generative model are used as the parameters of the initial bypass module. The parameters of the initial bypass module need to be adjusted in the subsequent training process, and the initial learning rate is 0.0004. The learning rate is used to control the adjustment amplitude of the parameters of the initial bypass module each time. The larger the learning rate, the larger the adjustment amplitude; conversely, the lower the learning rate, the lower the adjustment amplitude. The learning rate decreases as the number of iterative adjustments of the initial bypass module increases. For example, the learning rate of the first 10 rounds of iterative adjustments is 0.0004, and the learning rate of the 11th to 20th rounds of iterative adjustments is 0.00004.

[0102] (2) For any image-text pair in the training batch, obtain a third random noise image corresponding to a random number, superimpose (fuse) the scene sample image and the third random noise image to obtain a noisy sample image x, input the noisy sample image x into the image encoder, and the image encoder encodes the noisy sample image x to obtain an initial latent space representation Z, and perform T noise addition processing on the initial latent space representation Z to obtain a noisy latent space representation Z T .

[0103] (3) Input the text description into the text encoder to obtain the text representation τ, and then combine the text representation τ and the noisy latent space representation Z T Input into the denoising module and initial bypass module of the basic generation model (text representation as KV information), the noisy latent space representation Z T Under the constraint of KV information, perform T denoising processes. The first denoising process obtains the denoised latent space representation Z T-1 , until the output of the basic generation model after T denoising processes obtains the denoised latent space representation Z0, and the image decoder decodes the denoised latent space representation Z0 to obtain the predicted scene image.

[0104] (4) Calculate the loss function value corresponding to the training batch. Based on all scene sample images in the training batch and the corresponding predicted scene images, calculate the loss function value corresponding to the training batch. Exemplarily, the computer device may use the following formula (1) to calculate the loss function value corresponding to the training batch:

[0105]

[0106] In formula (1), MSE represents the loss function value corresponding to the training batch, F is the number of image-text pairs corresponding to the training batch, and y d is the pixel value of the scene sample image in the dth image-text pair, It is the pixel value of the predicted scene image obtained by the basic generation model and the initial bypass module based on the d-th image-text pair.

[0107] (5) Using the Stochastic Gradient Descent (SGD) method, the loss function value corresponding to the training batch is fed back to the gradient of the initial bypass module and the parameters are updated.

[0108] (6) Complete all batch training and end an iteration.

[0109] It should be noted that if Figure 8cAs shown in the figure, the downsampling unit of the upsampling network of the basic generation model, the downsampling unit of the downsampling network, and the bypass unit of the initial bypass module are all QKV units, where Q (Query) refers to to match others, indicating the information to be controlled (i.e., noisy latent space representation), K (Key) refers to to be matched, indicating the control information (i.e., the text representation corresponding to the description text), and V (Value) refers to information to be extracted, indicating the information of the output feature.

[0110] In this application, by saving scene-related knowledge into the trained bypass module, only one fine-tuning of the initial bypass model is required, and the bypass module can be embedded in different style models to generate images that meet the scene requirements and style requirements, thereby improving the utilization rate of the bypass module, eliminating the need for multiple fine-tuning, making it more convenient to use and saving resources. In other words, the bypass module obtained after one offline fine-tuning can meet the needs of image style conversion of various downstream style models, without the need to fine-tune a bypass module for each image style category, and is easy to apply.

[0111] See also Fig. 9 , is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. Fig. 9 As shown, the image generating device may include:

[0112] The first acquisition module 901 is used to acquire an image generation prompt text of the image to be generated; the image generation prompt text reflects the scene information, the first style information and the second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information;

[0113] A second acquisition module 902, configured to obtain a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information;

[0114] A first generating module 903 is used to call the bypass module and the style basic model, generate a prompt text according to the first random noise image and the image, and generate a pre-generated image matching both the scene information and the first style information;

[0115] The second generation module 904 is used to call the target style model to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image.

[0116] Optionally, the first generation module 903 calls the bypass module and the style basic model to generate a prompt text according to the first random noise image and the image, and generates a pre-generated image matching both the scene information and the first style information, including:

[0117] Obtaining a first noisy latent space representation corresponding to the first random noise image;

[0118] Calling the bypass module and the denoising module of the style base model, generating a prompt text according to the image, performing T denoising processes on the first noisy latent space representation, and obtaining a first denoised latent space representation; T is a positive integer;

[0119] The first denoised latent space representation is decoded to obtain a pre-generated image that matches both the scene information and the first style information.

[0120] Optionally, the denoising module of the style base model includes a first upsampling network and a first downsampling network;

[0121] The first generation module 903 calls the bypass module and the denoising module of the style base model, generates a prompt text according to the image, performs T denoising processes on the first noisy latent space representation, and obtains a first denoised latent space representation, including:

[0122] For the jth denoising process, the above image generates prompt text and input representation P j Input to the bypass module to obtain the output data of the bypass module in the j-th denoising process; the first input representation is the first noisy latent space representation, and j is a positive integer less than or equal to T;

[0123] The above image generates prompt text and the above input representation P j Input to the first down-sampling network to obtain the output data of the first down-sampling network after the j-th denoising process;

[0124] Input the output data of the bypass module after the j-th denoising process, the output data of the first downsampling network after the j-th denoising process, and the image generation prompt text into the first upsampling network to obtain the output data of the first upsampling network after the j-th denoising process;

[0125] The output data of the first upsampling network after the T-th denoising process is determined as the first denoising latent space representation.

[0126] Optionally, the bypass module includes S bypass units, the first downsampling network includes S first downsampling units, and S is a positive integer;

[0127] The first generating module 903 generates a prompt text and an input representation P from the image j Input to the first down-sampling network to obtain the output data of the first down-sampling network after the j-th denoising process, including:

[0128] The output data of the a-th bypass unit in the bypass module after the j-th denoising process, the output data of the first downsampling unit a after the j-th denoising process, and the image generation prompt text are input to the first downsampling unit a+1 to obtain the output data of the first downsampling unit a+1 after the j-th denoising process; a is a positive integer less than S;

[0129] The output data of the first down-sampling unit S after the j-th denoising process is determined as the output data of the first down-sampling network after the j-th denoising process; the input representation P j The prompt text generated from the above image is the input data of the first downsampling unit 1 in the above j-th denoising process.

[0130] Optionally, the second generation module 904 calls the target style model to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image, including:

[0131] Calling the edge map control module to perform L edge extractions on the pre-generated image to obtain L edge images reflecting the contour information in the pre-generated image; L is a positive integer;

[0132] Obtaining a second noisy latent space representation corresponding to the second random noise image;

[0133] Calling the denoising module of the target style model, performing L denoising processes on the second noisy latent space representation according to the L edge images and the image-generated prompt text, to obtain a second denoised latent space representation that matches the image-generated prompt text;

[0134] The second denoised latent space representation is decoded to obtain a target image that matches the image generation prompt text.

[0135] Optionally, the second generation module 904 calls the denoising module of the target style model, performs L denoising processes on the second noisy latent space representation according to the L edge images and the image generation prompt text, and obtains a second denoised latent space representation matching the image generation prompt text, including:

[0136] For the (i+1)th denoising process, obtain the denoised latent space representation i obtained by the above target style model in the i-th denoising process; the first denoised latent space representation is obtained according to the image-generated prompt text, the second noisy latent space representation and the first edge image, and i is a positive integer less than L;

[0137] Calling the denoising module of the target style model, generating a prompt text according to the i-th edge image and the image, and denoising the denoised latent space representation i to obtain a denoised latent space representation i+1;

[0138] The denoised latent space representation obtained by the L-th denoising process is determined as the second denoised latent space representation that matches the image generation prompt text.

[0139] Optionally, the second acquisition module 902 acquires a style basic model obtained by training the style sample image associated with the first style information, including:

[0140] Determine the style difference between the image style type indicated by the first style information and the real image style category;

[0141] If the style difference is greater than the difference threshold, a style generation model obtained by training the style sample image associated with the first style information is determined as the style base model;

[0142] If the style difference is less than or equal to the difference threshold, a basic generative model trained based on a common sample image is determined as a style basic model; the common sample image includes a style sample image associated with the first style information.

[0143] If it is necessary to explain, the image generating device in the present application can be used to execute the description of the above-mentioned image generating method in the corresponding embodiment of the previous text. When the image generating device executes the description of the image generating method of the previous text, the beneficial effects brought about can refer to the beneficial effects described in the embodiment corresponding to the image generating method of the previous text, and will not be repeated here.

[0144] In the present application, the style models (i.e., the style base model and the target style model) used for image style conversion and the bypass module used to generate the scene image are independently trained, i.e., the style base model is trained based on the style sample images associated with the first style information, the target style model is trained based on the style sample images associated with the second style information, and the bypass module is trained based on the scene sample images associated with the scene information. This can avoid the problem of too few sample images that meet both the scene information and the style information, resulting in low image generation quality, thereby improving the image generation quality. In practical applications, by embedding the bypass module into a style base model with better generalization performance to generate a pre-generated image that conforms to the semantic information, the semantic information refers to the scene information and the first style information, and through the target style model, the image style conversion is performed with the pre-generated image as the control information, and a target image that matches the image generation prompt text is obtained, so that the target image meets the scene requirements while meeting the image style requirements, thereby improving the image generation quality.

[0145] See also Fig.10 , is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. Fig.10 As shown, the image generating device may include:

[0146] The acquisition module 1101 is used to acquire N image-text pairs associated with the above scene information, an initial bypass module, and a basic generation model trained based on a general sample image; an image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1;

[0147] A noise processing module 1102 is used to call the diffusion module of the basic generation model to perform noise processing on the scene sample images in the N image-text pairs to obtain N third noisy latent space representations;

[0148] A denoising processing module 1103 is used to call the denoising module of the basic generation model and the initial bypass module, and perform denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images;

[0149] The adjustment module 1104 is used to adjust the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module; the bypass module is used in the method described above.

[0150] Optionally, the denoising processing module 1103 calls the denoising module of the basic generation model and the initial bypass module, and performs denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images, including:

[0151] Calling the denoising module of the basic generation model and the initial bypass module, and performing T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N third denoised latent space representations;

[0152] The N third denoised latent space representations are decoded to obtain N predicted scene images.

[0153] Optionally, the denoising module of the above basic generation model includes a second upsampling network and a second downsampling network;

[0154] The denoising processing module 1103 calls the denoising module of the basic generation model and the initial bypass module, and performs T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N third denoised latent space representations, including:

[0155] For the jth denoising process, the description text k in the kth image-text pair and the input representation Q j Input to the above-mentioned initial bypass module to obtain the output data of the above-mentioned initial bypass module in the above-mentioned j-th denoising process; the first input representation is the third noisy latent space representation k, j is a positive integer less than or equal to T, and k is a positive integer less than or equal to N;

[0156] The above description text k and the above input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process;

[0157] Input the output data of the initial bypass module after the j-th denoising process, the output data of the second down-sampling network after the j-th denoising process, and the description text k into the second up-sampling network to obtain the output data of the second up-sampling network after the j-th denoising process;

[0158] The output data of the T-th denoising process of the second upsampling network is determined as the third denoising latent space representation associated with the k-th image-text pair.

[0159] Optionally, the initial bypass module includes S bypass units, the second downsampling network includes S second downsampling units, and S is a positive integer;

[0160] The denoising module 1103 combines the description text k and the input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process, including:

[0161] The output data of the a-th bypass unit in the initial bypass module after the j-th denoising process, the output data of the second downsampling unit a after the j-th denoising process and the description text k are input into the second downsampling unit a+1 to obtain the output data of the second downsampling unit a+1 after the j-th denoising process; a is a positive integer less than S;

[0162] The output data of the second down-sampling unit S after the j-th denoising process is determined as the output data of the second down-sampling network after the j-th denoising process; the input representation Q j And the above description text k is the input data of the second down-sampling unit 1 in the above j-th denoising process.

[0163] Optionally, the adjustment module 1104 adjusts the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module, including:

[0164] Determining a loss function value of the initial bypass module according to a difference between the N predicted scene images and the sample scene images in the N image-text pairs;

[0165] According to the above loss function value, the parameters of the above initial bypass module are adjusted for multiple rounds of iterations to obtain a trained bypass module.

[0166] Optionally, the acquisition module 1101 acquires N image-text pairs associated with the scene information, including:

[0167] Acquire N scene sample images associated with the scene information, and initial description texts corresponding to the N scene sample images respectively;

[0168] Extracting scene content information and scene atmosphere information corresponding to the scene sample image c from the initial description text corresponding to the scene sample image c; c is a positive integer less than or equal to N;

[0169] Determine the scene content information and scene atmosphere information corresponding to the scene sample image c as a description text of the scene sample image c;

[0170] N image-text pairs are generated according to the N scene sample images and the description texts of the N scene sample images.

[0171] If it is necessary to explain, the image generating device in the present application can be used to execute the description of the above-mentioned image generating method in the corresponding embodiment of the previous text. When the image generating device executes the description of the image generating method of the previous text, the beneficial effects brought about can refer to the beneficial effects described in the embodiment corresponding to the image generating method of the previous text, and will not be repeated here.

[0172] In this application, by saving scene-related knowledge into the trained bypass module, only one fine-tuning of the initial bypass model is required, and the bypass module can be embedded in different style models to generate images that meet the scene requirements and style requirements, thereby improving the utilization rate of the bypass module, eliminating the need for multiple fine-tuning, making it more convenient to use and saving resources. In other words, the bypass module obtained after one offline fine-tuning can meet the needs of image style conversion of various downstream style models, without the need to fine-tune a bypass module for each image style category, and is easy to apply.

[0173] See also Fig.11 , is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.11 As shown, the above-mentioned computer device 1000 may refer to a terminal or a server, including: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, in some embodiments, the user interface 1003 may include a display screen (DiSPlay), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatileMeMory), such as at least one disk storage. The memory 1005 may also be optionally at least one storage device away from the aforementioned processor 1001. As Fig.11 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a computer application program.

[0174] exist Fig.11 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface; and the processor 1001 can be used to call a computer application stored in the memory 1005 to achieve:

[0175] Obtaining image generation prompt text of the image to be generated; the image generation prompt text reflects scene information, first style information and second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information;

[0176] Acquire a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information;

[0177] Calling the bypass module and the style basic model, generating a prompt text according to the first random noise image and the image, and generating a pre-generated image that matches both the scene information and the first style information;

[0178] The target style model is called to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image.

[0179] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to call the bypass module and the style base model, generate a prompt text according to the first random noise image and the image, and generate a pre-generated image matching both the scene information and the first style information, including:

[0180] Obtaining a first noisy latent space representation corresponding to the first random noise image;

[0181] Calling the bypass module and the denoising module of the style base model, generating a prompt text according to the image, and performing T denoising processes on the first noisy latent space representation to obtain a first denoised latent space representation; T is a positive integer;

[0182] The first denoised latent space representation is decoded to obtain a pre-generated image that matches both the scene information and the first style information.

[0183] Optionally, the denoising module of the style base model includes a first upsampling network and a first downsampling network;

[0184] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to call the bypass module and the denoising module of the style base model, generate a prompt text according to the image, perform T denoising processes on the first noisy latent space representation, and obtain a first denoised latent space representation, including:

[0185] For the jth denoising process, the above image generates prompt text and input representation P j Input to the bypass module to obtain the output data of the bypass module in the j-th denoising process; the first input representation is the first noisy latent space representation, and j is a positive integer less than or equal to T;

[0186] The above image generates prompt text and the above input representation P j Input to the first down-sampling network to obtain the output data of the first down-sampling network after the j-th denoising process;

[0187] Input the output data of the bypass module after the j-th denoising process, the output data of the first downsampling network after the j-th denoising process, and the image generation prompt text into the first upsampling network to obtain the output data of the first upsampling network after the j-th denoising process;

[0188] The output data of the first upsampling network after the T-th denoising process is determined as the first denoising latent space representation.

[0189] Optionally, the bypass module includes S bypass units, the first down-sampling network includes S first down-sampling units, and S is a positive integer.

[0190] Optionally, the processor 1001 is used to call the computer application stored in the memory 1005 to realize the generation of the prompt text and the input representation P from the above image. j Input to the first down-sampling network to obtain the output data of the first down-sampling network after the j-th denoising process, including:

[0191] The output data of the a-th bypass unit in the bypass module after the j-th denoising process, the output data of the first downsampling unit a after the j-th denoising process, and the image generation prompt text are input to the first downsampling unit a+1 to obtain the output data of the first downsampling unit a+1 after the j-th denoising process; a is a positive integer less than S;

[0192] The output data of the first down-sampling unit S after the j-th denoising process is determined as the output data of the first down-sampling network after the j-th denoising process; the input representation P j The prompt text generated from the above image is the input data of the first downsampling unit 1 in the above j-th denoising process.

[0193] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to call the target style model, and generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text, and the pre-generated image, including:

[0194] Calling the edge map control module to perform L edge extractions on the pre-generated image to obtain L edge images reflecting the contour information in the pre-generated image; L is a positive integer;

[0195] Obtaining a second noisy latent space representation corresponding to the second random noise image;

[0196] Calling the denoising module of the target style model, performing L denoising processes on the second noisy latent space representation according to the L edge images and the image-generated prompt text, to obtain a second denoised latent space representation that matches the image-generated prompt text;

[0197] The second denoised latent space representation is decoded to obtain a target image that matches the image generation prompt text.

[0198] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to implement calling the denoising module of the target style model, and according to the L edge images and the image generation prompt text, perform L denoising processes on the second noisy latent space representation to obtain a second denoised latent space representation matching the image generation prompt text, including:

[0199] For the (i+1)th denoising process, obtain the denoised latent space representation i obtained by the above target style model in the i-th denoising process; the first denoised latent space representation is obtained according to the image-generated prompt text, the second noisy latent space representation and the first edge image, and i is a positive integer less than L;

[0200] Calling the denoising module of the target style model, generating a prompt text according to the i-th edge image and the image, and denoising the denoised latent space representation i to obtain a denoised latent space representation i+1;

[0201] The denoised latent space representation obtained by the L-th denoising process is determined as the second denoised latent space representation that matches the image generation prompt text.

[0202] Optionally, the processor 1001 is configured to call a computer application stored in the memory 1005 to obtain a style base model trained based on the style sample image associated with the first style information, including:

[0203] Determine the style difference between the image style type indicated by the first style information and the real image style category;

[0204] If the style difference is greater than the difference threshold, a style generation model obtained by training the style sample image associated with the first style information is determined as the style base model;

[0205] If the style difference is less than or equal to the difference threshold, a basic generative model trained based on a common sample image is determined as a style basic model; the common sample image includes a style sample image associated with the first style information.

[0206] Optionally, the processor 1001 may be configured to call a computer application stored in the memory 1005 to implement:

[0207] Obtain N image-text pairs associated with the above scene information, an initial bypass module, and a basic generation model trained based on a general sample image; an image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1;

[0208] Calling the diffusion module of the above basic generation model to perform noise processing on the scene sample images in the above N image-text pairs to obtain N third noisy latent space representations;

[0209] Calling the denoising module of the basic generation model and the initial bypass module, and performing denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N predicted scene images;

[0210] According to the N predicted scene images and the sample scene images in the N image-text pairs, the parameters of the initial bypass module are adjusted to obtain a trained bypass module; the bypass module is used in the above method.

[0211] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to implement calling the denoising module of the basic generation model and the initial bypass module, and perform denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images, including:

[0212] Calling the denoising module of the basic generation model and the initial bypass module, and performing T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N third denoised latent space representations;

[0213] The N third denoised latent space representations are decoded to obtain N predicted scene images.

[0214] Optionally, the denoising module of the above basic generation model includes a second upsampling network and a second downsampling network;

[0215] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to implement calling the denoising module and the initial bypass module of the basic generation model, and perform T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N third denoised latent space representations, including:

[0216] For the jth denoising process, the description text k in the kth image-text pair and the input representation Q j Input to the above-mentioned initial bypass module to obtain the output data of the above-mentioned initial bypass module in the above-mentioned j-th denoising process; the first input representation is the third noisy latent space representation k, j is a positive integer less than or equal to T, and k is a positive integer less than or equal to N;

[0217] The above description text k and the above input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process;

[0218] Input the output data of the initial bypass module after the j-th denoising process, the output data of the second down-sampling network after the j-th denoising process, and the description text k into the second up-sampling network to obtain the output data of the second up-sampling network after the j-th denoising process;

[0219] The output data of the T-th denoising process of the second upsampling network is determined as the third denoising latent space representation associated with the k-th image-text pair.

[0220] Optionally, the initial bypass module includes S bypass units, the second downsampling network includes S second downsampling units, and S is a positive integer;

[0221] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to implement the above description text k and the above input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process, including:

[0222] The output data of the a-th bypass unit in the initial bypass module after the j-th denoising process, the output data of the second downsampling unit a after the j-th denoising process and the description text k are input into the second downsampling unit a+1 to obtain the output data of the second downsampling unit a+1 after the j-th denoising process; a is a positive integer less than S;

[0223] The output data of the second down-sampling unit S after the j-th denoising process is determined as the output data of the second down-sampling network after the j-th denoising process; the input representation Q j And the above description text k is the input data of the second down-sampling unit 1 in the above j-th denoising process.

[0224] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to adjust the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module, including:

[0225] Determining a loss function value of the initial bypass module according to a difference between the N predicted scene images and the sample scene images in the N image-text pairs;

[0226] According to the above loss function value, the parameters of the above initial bypass module are adjusted for multiple rounds of iterations to obtain a trained bypass module.

[0227] Optionally, the processor 1001 is used to call a computer application stored in the memory 1005 to obtain N image-text pairs associated with the above scene information, including:

[0228] Acquire N scene sample images associated with the scene information, and initial description texts corresponding to the N scene sample images respectively;

[0229] Extracting scene content information and scene atmosphere information corresponding to the scene sample image c from the initial description text corresponding to the scene sample image c; c is a positive integer less than or equal to N;

[0230] Determine the scene content information and scene atmosphere information corresponding to the scene sample image c as a description text of the scene sample image c;

[0231] N image-text pairs are generated according to the N scene sample images and the description texts of the N scene sample images.

[0232] In the present application, the style models (i.e., the style base model and the target style model) used for image style conversion and the bypass module used to generate the scene image are independently trained, i.e., the style base model is trained based on the style sample images associated with the first style information, the target style model is trained based on the style sample images associated with the second style information, and the bypass module is trained based on the scene sample images associated with the scene information. This can avoid the problem of too few sample images that meet both the scene information and the style information, resulting in low image generation quality, thereby improving the image generation quality. In practical applications, by embedding the bypass module into a style base model with better generalization performance to generate a pre-generated image that conforms to the semantic information, the semantic information refers to the scene information and the first style information, and through the target style model, the image style conversion is performed with the pre-generated image as the control information, and a target image that matches the image generation prompt text is obtained, so that the target image meets the scene requirements while meeting the image style requirements, thereby improving the image generation quality.

[0233] It should be understood that the computer device described in the embodiments of the present application can execute the description of the above-mentioned image generation method in the corresponding embodiments above, and can also execute the description of the above-mentioned image generation device in the corresponding embodiments above, which will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated.

[0234] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the above-mentioned computer-readable storage medium stores a computer program executed by the image generation device mentioned above, and the above-mentioned computer program includes program instructions. When the above-mentioned processor executes the above-mentioned program instructions, it can execute the description of the above-mentioned image generation method in the corresponding embodiment above, so it will not be repeated here. In addition, the description of the beneficial effects of using the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0235] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on at least two computer devices at one location for execution, or executed on at least two computer devices distributed at at least two locations and interconnected through a communication network. At least two computer devices distributed at at least two locations and interconnected through a communication network may constitute a blockchain network.

[0236] The computer-readable storage medium may be the image generation device provided in any of the aforementioned embodiments or the central storage unit of the computer device, such as a hard disk or a memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SMART Media card, SMC), a secure digital (Secure digital, SD) card, a flash memory card (flaSh card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both the central storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0237] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish between contents in different media, rather than to describe a specific order. In addition, the terms "including" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other step units inherent to these processes, methods, devices, products, or equipment.

[0238] The data collection and processing in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in practice, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0239] The embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the description of the above-mentioned image generation method and decoding method in the above-mentioned corresponding embodiment, so it will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the embodiment of the computer program product involved in the present application, please refer to the description of the method embodiment of the present application.

[0240] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0241] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable network connection device to generate a machine, so that the instructions executed by the processor of the computer or other programmable network connection device generate instructions for implementing the process in the process. Figure 1 Schematic diagram of one or more processes and / or structures Figure 1These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable network-connected device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable network connected device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0242] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image generation method, characterized in that: include: Get the image generation prompt text of the image to be generated; The image generation prompt text reflects the scene information, the first style information and the second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information; Acquire a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information; Calling the bypass module and the style base model, generating prompt text according to the first random noise image and the image, and generating a pre-generated image that matches both the scene information and the first style information; The target style model is called to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image.

2. The method according to claim 1, characterized in that The calling of the bypass module and the style base model, generating a prompt text according to the first random noise image and the image, and generating a pre-generated image matching both the scene information and the first style information, comprises: Obtaining a first noisy latent space representation corresponding to the first random noise image; Calling the bypass module and the denoising module of the style base model, generating a prompt text according to the image, and performing T denoising processes on the first noisy latent space representation to obtain a first denoised latent space representation; T is a positive integer; The first denoised latent space representation is decoded to obtain a pre-generated image that matches both the scene information and the first style information.

3. The method according to claim 2, characterized in that The denoising module of the style base model includes a first upsampling network and a first downsampling network; The calling of the bypass module and the denoising module of the style base model, generating a prompt text according to the image, and performing T denoising processes on the first noisy latent space representation to obtain a first denoised latent space representation, includes: For the jth denoising process, the image generates prompt text and input representation P j Input to the bypass module to obtain the output data of the bypass module after the j-th denoising process; the first input representation is the first noisy latent space representation, and j is a positive integer less than or equal to T; The image generates a prompt text and the input representation P j Input to the first down-sampling network to obtain output data of the first down-sampling network after the j-th denoising process; Input the output data of the bypass module after the j-th denoising process, the output data of the first down-sampling network after the j-th denoising process, and the image generation prompt text into the first up-sampling network to obtain the output data of the first up-sampling network after the j-th denoising process; The output data of the first upsampling network after the T-th denoising process is determined as the first denoising latent space representation.

4. The method according to claim 3, characterized in that The bypass module includes S bypass units, the first down-sampling network includes S first down-sampling units, and S is a positive integer; The image generates a prompt text and an input representation P j Inputting into the first down-sampling network, obtaining the output data of the first down-sampling network after the j-th denoising process, comprises: The output data of the ath bypass unit in the bypass module after the jth denoising process, the output data of the first downsampling unit a after the jth denoising process, and the image generation prompt text are input to the first downsampling unit a+1 to obtain the output data of the first downsampling unit a+1 after the jth denoising process; a is a positive integer less than S; The output data of the first down-sampling unit S in the j-th denoising process is determined as the output data of the first down-sampling network in the j-th denoising process; the input representation P j The image generation prompt text is the input data of the first downsampling unit 1 in the j-th denoising process.

5. The method according to claim 1, characterized in that The calling of the target style model to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image includes: Calling the edge map control module to perform L edge extractions on the pre-generated image to obtain L edge images reflecting contour information in the pre-generated image; L is a positive integer; Obtaining a second noisy latent space representation corresponding to the second random noise image; Calling the denoising module of the target style model, performing L denoising processes on the second noisy latent space representation according to the L edge images and the image generation prompt text, to obtain a second denoised latent space representation that matches the image generation prompt text; The second denoised latent space representation is decoded to obtain a target image that matches the image generation prompt text.

6. The method according to claim 5, characterized in that The calling of the denoising module of the target style model, performing L denoising processes on the second noisy latent space representation according to the L edge images and the image generation prompt text, to obtain a second denoised latent space representation matching the image generation prompt text, comprises: For the (i+1)th denoising process, obtaining a denoised latent space representation i of the target style model obtained in the i-th denoising process; the first denoised latent space representation is obtained according to the image-generated prompt text, the second noisy latent space representation and the first edge image, and i is a positive integer less than L; Calling the denoising module of the target style model, generating a prompt text according to the i-th edge image and the image, and performing denoising processing on the denoised latent space representation i to obtain a denoised latent space representation i+1; The denoised latent space representation obtained by the L-th denoising process is determined as a second denoised latent space representation that matches the image generation prompt text.

7. The method according to claim 1, characterized in that The acquiring of a basic style model obtained by training based on a style sample image associated with the first style information includes: Determine a style difference between the image style type indicated by the first style information and a real image style category; If the style difference is greater than the difference threshold, determining a style generation model obtained by training the style sample image associated with the first style information as the style basic model; If the style difference is less than or equal to the difference threshold, a basic generative model trained based on a universal sample image is determined as a style basic model; the universal sample image includes a style sample image associated with the first style information.

8. An image generation method, characterized in that: include: Obtaining N image-text pairs associated with the scene information, an initial bypass module, and a basic generation model trained based on a general sample image; An image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1; Calling the diffusion module of the basic generation model to perform noise processing on the scene sample images in the N image-text pairs to obtain N third noisy latent space representations; Calling the denoising module and the initial bypass module of the basic generation model to denoise the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images; According to the N predicted scene images and the sample scene images in the N image-text pairs, the parameters of the initial bypass module are adjusted to obtain a trained bypass module; the bypass module is used to implement the method as described in any one of claims 1-7.

9. The method according to claim 8, characterized in that The calling of the denoising module of the basic generation model and the initial bypass module performs denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images, including: Calling the denoising module and the initial bypass module of the basic generation model, and performing T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N third denoised latent space representations; The N third denoised latent space representations are decoded to obtain N predicted scene images.

10. The method according to claim 9, characterized in that The denoising module of the basic generation model includes a second upsampling network and a second downsampling network; The calling of the denoising module and the initial bypass module of the basic generation model, performing T denoising processes on the N third noisy latent space representations according to the description texts in the N image-text pairs, to obtain N third denoised latent space representations, includes: For the jth denoising process, the description text k in the kth image-text pair and the input representation Q j Input to the initial bypass module to obtain the output data of the initial bypass module in the j-th denoising process; the first input representation is the third noisy latent space representation k, j is a positive integer less than or equal to T, and k is a positive integer less than or equal to N; The description text k and the input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process; Input the output data of the initial bypass module after the j-th denoising process, the output data of the second down-sampling network after the j-th denoising process, and the description text k into the second up-sampling network to obtain the output data of the second up-sampling network after the j-th denoising process; The output data of the T-th denoising process of the second upsampling network is determined as the third denoising latent space representation associated with the k-th image-text pair.

11. The method according to claim 10, characterized in that The initial bypass module includes S bypass units, the second downsampling network includes S second downsampling units, and S is a positive integer; The description text k and the input representation Q j Input to the second down-sampling network to obtain the output data of the second down-sampling network after the j-th denoising process, including: The output data of the ath bypass unit in the initial bypass module after the jth denoising process, the output data of the second downsampling unit a after the jth denoising process and the description text k are input into the second downsampling unit a+1 to obtain the output data of the second downsampling unit a+1 after the jth denoising process; a is a positive integer less than S; The output data of the second down-sampling unit S in the j-th denoising process is determined as the output data of the second down-sampling network in the j-th denoising process; the input representation Q j And the description text k is the input data of the second down-sampling unit 1 in the j-th denoising process.

12. The method according to claim 8, characterized in that The adjusting the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module includes: Determining a loss function value of the initial bypass module according to a difference between the N predicted scene images and the sample scene images in the N image-text pairs; According to the loss function value, the parameters of the initial bypass module are adjusted for multiple rounds of iterations to obtain a trained bypass module.

13. The method according to claim 8, characterized in that The acquiring N image-text pairs associated with the scene information includes: Acquire N scene sample images associated with the scene information, and initial description texts respectively corresponding to the N scene sample images; Extracting scene content information and scene atmosphere information corresponding to the scene sample image c from the initial description text corresponding to the scene sample image c; c is a positive integer less than or equal to N; Determine the scene content information and scene atmosphere information corresponding to the scene sample image c as a description text of the scene sample image c; N image-text pairs are generated according to the N scene sample images and the description texts of the N scene sample images.

14. An image generating device, characterized in that: include: A first acquisition module is used to acquire an image generation prompt text of an image to be generated; The image generation prompt text reflects the scene information, the first style information and the second style information of the image to be generated, and the image style category indicated by the second style information is a subcategory of the image style category indicated by the first style information; A second acquisition module, configured to obtain a style base model trained based on the style sample images associated with the first style information, a bypass module trained based on the scene sample images associated with the scene information, and a target style model trained based on the style sample images associated with the second style information; A first generating module, configured to call the bypass module and the style base model, generate a prompt text according to a first random noise image and the image, and generate a pre-generated image matching both the scene information and the first style information; The second generation module is used to call the target style model to generate a target image matching the image generation prompt text according to the second random noise image, the image generation prompt text and the pre-generated image.

15. An image generating device, characterized in that: include: A first acquisition module, used to acquire N image-text pairs associated with the scene information, an initial bypass module, and a basic generation model obtained by training based on a general sample image; An image-text pair includes a scene sample image and a description text of the scene sample image; N is a positive integer greater than 1; A noise processing module, used for calling the diffusion module of the basic generation model to perform noise processing on the scene sample images in the N image-text pairs to obtain N third noisy latent space representations; A denoising processing module, used to call the denoising module of the basic generation model and the initial bypass module, and perform denoising processing on the N third noisy latent space representations according to the description texts in the N image-text pairs to obtain N predicted scene images; An adjustment module is used to adjust the parameters of the initial bypass module according to the N predicted scene images and the sample scene images in the N image-text pairs to obtain a trained bypass module; the bypass module is used to implement the method described in any one of claims 1-7.

16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 13 are implemented.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

18. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 13 are implemented.

Images