Image generation method, device, electronic device and computer readable storage medium

Abstract

The present application is applicable to the field of image processing technology, and in particular, to an image generation method, device, electronic device and computer-readable storage medium. In this method, when performing AI painting, the electronic device can obtain hand-drawn images and text content, and input the hand-drawn images and text content into a trained image generation model for processing to obtain a generated image output by the image generation model; wherein the image generation model includes a first branch and a second branch, the first branch is used to extract features from the hand-drawn image, obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch, the second branch is used to process the hand-drawn features and text content to obtain a generated image, and the second branch is a stable diffusion model. That is, the image generation model in this application can accurately generate the image desired by the user based on the hand-drawn image drawn by the user and the input text content, which can meet the actual needs of the user and improve the user experience.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular, relates to an image generation method, device, electronic device and computer-readable storage medium. Background Art

[0002] Artificial intelligence (AI) technology can be widely used in people's daily lives, bringing great convenience and fun to people's daily lives. For example, it can be applied to painting scenes to realize AI painting. That is, AI painting can refer to image generation through AI technology. However, current AI painting is generally based on the text description entered by the user to generate the image corresponding to the text description, that is, the user needs to enter an accurate text description to generate the image the user wants, resulting in a poor user experience and unable to meet the actual needs of the user. Summary of the invention

[0003] The embodiments of the present application provide an image generation method, device, electronic device and computer-readable storage medium, which can accurately generate the image desired by the user, meet the actual needs of the user, and improve the user experience.

[0004] In a first aspect, an embodiment of the present application provides an image generation method, comprising:

[0005] Get hand-drawn images and text content;

[0006] Inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model;

[0007] Among them, the image generation model includes a first branch and a second branch. The first branch is used to extract features of the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch. The second branch is used to process the hand-drawn features and the text content to obtain the generated image. The second branch is a stable diffusion model.

[0008] In the image generation method provided above, when performing AI painting, the user can draw a sketch corresponding to the desired image (i.e., a hand-drawn image), and can input text content (i.e., a text description corresponding to the desired image). After the electronic device obtains the hand-drawn image and text content, the hand-drawn image and text content can be input into a trained image generation model for processing to obtain a generated image output by the image generation model; wherein the image generation model may include a first branch and a second branch, the first branch is used to extract features from the hand-drawn image, obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch, the second branch is used to process the hand-drawn features and text content to obtain a generated image, and the second branch is a stable diffusion model. That is, in an embodiment of the present application, the image generation model can accurately generate the image desired by the user based on the hand-drawn image drawn by the user and the input text content, which can meet the actual needs of the user and improve the user experience.

[0009] In some embodiments, the first branch includes a preprocessing module and a feature extraction module, and the inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model includes:

[0010] Inputting the hand-drawn image and the text content into the image generation model;

[0011] Performing dimensionality reduction processing on the hand-drawn image through the preprocessing module in the first branch to obtain an intermediate image corresponding to the hand-drawn image;

[0012] Performing feature extraction on the intermediate image by using the feature extraction module in the first branch to obtain hand-drawn features corresponding to the hand-drawn image, and inputting the hand-drawn features into the second branch;

[0013] The hand-drawn features and the text content are processed through the second branch to obtain the generated image output by the image generation model.

[0014] In a possible implementation, the feature extraction module includes a first network layer, a first downsampling layer, a second network layer, a second downsampling layer, a third network layer, a third downsampling layer and a fourth network layer, and the feature extraction module in the first branch performs feature extraction on the intermediate image to obtain hand-drawn features corresponding to the hand-drawn image, and inputs the hand-drawn features into the second branch, including:

[0015] Performing feature extraction on the intermediate image through the first network layer to obtain a first hand-drawn feature corresponding to the hand-drawn image, and inputting the first hand-drawn feature into the second branch;

[0016] Downsampling the first hand-drawn feature through the first downsampling layer, processing the downsampled first hand-drawn feature through the second network layer to obtain a second hand-drawn feature corresponding to the hand-drawn image, and inputting the second hand-drawn feature into the second branch;

[0017] Downsampling the second hand-drawn feature through the second downsampling layer, processing the downsampled second hand-drawn feature through the third network layer to obtain a third hand-drawn feature corresponding to the hand-drawn image, and inputting the third hand-drawn feature into the second branch;

[0018] The third hand-drawn feature is downsampled by the third downsampling layer, and the downsampled third hand-drawn feature is processed by the fourth network layer to obtain a fourth hand-drawn feature corresponding to the hand-drawn image, and the fourth hand-drawn feature is input into the second branch.

[0019] In some other embodiments, the image generation model further includes a third branch, and the third branch is a low-rank adaptive LoRA model;

[0020] The third branch is used to fine-tune the image generation model according to a preset image style, so that the image generation model outputs a generated image of the preset image style.

[0021] In some embodiments, the method further comprises:

[0022] Obtaining preset detailed content, wherein the detailed content is content in text form;

[0023] The step of inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model includes:

[0024] The hand-drawn image, the text content and the detail content are input into the image generation model for processing to obtain the generated image output by the image generation model.

[0025] In some embodiments, before inputting the hand-drawn image and the text content into a trained image generation model for processing, the method further includes:

[0026] The image generation model is trained according to multiple groups of training data to obtain the trained image generation model, wherein each group of training data includes a sample image, a training hand-drawn image drawn based on the sample image, and training text content corresponding to the sample image.

[0027] In one example, the loss function used by the image generation model during training is:

[0028]

[0029] Among them, L simple is the loss function, t is the time, c t is the training text content corresponding to the sample image, F c is the hand-drawn feature corresponding to the training hand-drawn image, x 0 is the sample image, x t is the generated image at time t, ε is the actual noise added, and ε θ (x t ,c t ,t,F c ) is the predicted noise at time t.

[0030] In a second aspect, an embodiment of the present application provides an image generating device, including:

[0031] A hand-drawn image acquisition module is used to acquire hand-drawn images and text content;

[0032] An image generation module, used for inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model;

[0033] Among them, the image generation model includes a first branch and a second branch. The first branch is used to extract features of the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch. The second branch is used to process the hand-drawn features and the text content to obtain the generated image. The second branch is a stable diffusion model.

[0034] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the image generation method described in any one of the first aspects above.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the computer implements the image generation method described in any one of the first aspects above.

[0036] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the image generation method described in any one of the first aspects above.

[0037] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 is an example diagram of a diffusion model;

[0040] Figure 2 is an example diagram of a stable diffusion model;

[0041] Figure 3 It is a structural schematic diagram of the image generation model provided in the embodiment of the present application;

[0042] Figure 4 It is a flowchart of the image generation method provided in the embodiment of the present application;

[0043] Figure 5 and Figure 6 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0044] Figure 7 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application;

[0045] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0047] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0048] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0049] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0050] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0051] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0052] AI technology can be widely used in people's daily lives, bringing great convenience and fun to people's daily lives. For example, it can be applied to painting scenes to achieve AI painting. That is, AI painting can refer to image generation through AI technology.

[0053] For example, in AI painting, a stable diffusion model can be used to generate images related to the text description based on the text description input by the user. When generating images through stable diffusion, if the text description input by the user is inaccurate, the generated image is not what the user wants. At this time, the user needs to constantly adjust the text description to finally generate the image the user wants.

[0054] That is, when generating images through stable diffusion, users are required to enter accurate text descriptions in order to generate the images they want. However, when users are unable to describe the images they want through text, or the text descriptions are inaccurate, the images generated through stable diffusion will not be what the users want, resulting in a poor user experience and failure to meet the actual needs of users.

[0055] To solve the above problems, the embodiments of the present application provide an image generation method, device, electronic device and computer-readable storage medium. In the method, when performing AI painting, the user can draw a sketch corresponding to the desired image (i.e., a hand-drawn image), and can input text content (i.e., a text description corresponding to the desired image). After the electronic device obtains the hand-drawn image and the text content, the hand-drawn image and the text content can be input into the trained image generation model for processing to obtain a generated image output by the image generation model; wherein the image generation model may include a first branch and a second branch, the first branch is used to extract features from the hand-drawn image, obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch, the second branch is used to process the hand-drawn features and text content to obtain a generated image, and the second branch is a stable diffusion model. That is, in the embodiment of the present application, the image generation model can accurately generate the image desired by the user based on the hand-drawn image drawn by the user and the input text content, which can meet the actual needs of the user, improve the user experience, and have strong ease of use and practicality.

[0056] The image generation method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), smart large screens, servers, etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0057] The following first explains the key technical terms that may be involved in the embodiments of the present application.

[0058] 1. Diffusion model

[0059] See also Figure 1 , Figure 1 An example diagram of a diffusion model is shown.

[0060] like Figure 1As shown in Figure 1, the diffusion model is a generative model that can include diffusion process and inverse diffusion process.

[0061] The diffusion process can refer to the process from x 0 to x T The noise adding process. That is, the diffusion process can be from a real sample image x 0 Start by moving step by step towards the sample image x 0 Add noise to generate a series of intermediate states (or the generated image at a certain moment) x 1 , x 2 , ..., x T , and finally reach a noise x close to the standard Gaussian distribution T That is, the diffusion process can make the sample image more and more blurred by adding Gaussian noise in each step t. For example, the diffusion model can add Gaussian noise to the generated image of the previous step (i.e., time t-1) to obtain the generated image q(x) in this step (i.e., time t). t |x t-1 ).

[0062] The reverse diffusion process can be referred to as the process from x T to x 0 The noise reduction process. That is, the inverse diffusion process can be reduced from the noise x T Start by removing noise step by step to generate the final sample image x 0 Among them, each step of the reverse diffusion process can be represented by a learned network model p θ (x t-1 |x t ), the goal is to reverse the process of adding noise as accurately as possible. It should be understood that image generation can refer to the inverse diffusion process, that is, by gradually predicting the noise ε θ (x T ,t) to gradually generate x 0 process.

[0063] It should be noted that the loss function of the diffusion model can generally be:

[0064]

[0065] Among them, L simple is the loss function, t is the time, x 0 is the final generated image, x t is the generated image at time t, ε is the actual noise added, and ε θ (x t ,t) is the predicted noise at time t.

[0066] For the image generation at time t-1, the network model needs to predict the noise ε θ (x t ,t). In the prediction, the noise ε θ (x t ,t), the generated image x at time t-1 can be obtained by the following formula t-1 , and the final generated image x can be obtained by repeated iteration of the following formula 0 .

[0067]

[0068] Among them, α t =1-β t , β t is the noise variance imposed by the diffusion process, β t is a known value, σ t is the noise standard deviation, z refers to the noise of the standard normal distribution.

[0069] 2. Stable diffusion model.

[0070] See also Figure 2 , Figure 2 An example diagram of a steady-diffusion model is shown.

[0071] like Figure 2 As shown, the stable diffusion model no longer directly predicts the noise of the generated image, but predicts the noise of the latent space of the intermediate process, that is, the noise of the compressed image 210. The stable diffusion model may include a trained encoder (image encoder) 220 and a trained decoder (image decoder) 230. The stable diffusion model can encode the full-size original image into low-dimensional latent data (i.e., the compressed image 210) through the encoder 220, and can process the latent data to obtain the latent features of the original image in the latent space, and can input the latent features of the latent space into the decoder 230. The decoder 230 can decode the latent features of the latent space to reconstruct the image, thereby obtaining the generated image.

[0072] Among them, the stable diffusion model uses text guidance when generating images, so that the generated images are associated with the input text description through text guidance.

[0073] It should be noted that the above description of the diffusion model and the stable diffusion model is only for illustrative explanation and should not be understood as a limitation on the embodiments of the present application. In the embodiments of the present application, the specific content of the diffusion model and the specific content of the stable diffusion model can refer to the relevant description in the prior art.

[0074] The image generation model provided in the embodiments of the present application will be described in detail below.

[0075] See also Figure 3 , Figure 3 A schematic diagram of the structure of the image generation model provided in an embodiment of the present application is shown.

[0076] like Figure 3 As shown, the image generation model may include a first branch 310 and a second branch 320. The first branch 310 may be used to extract features from the hand-drawn image, obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch 320.

[0077] The second branch 320 can be used to process the hand-drawn features and text content to obtain a generated image. The second branch 320 can be a stable diffusion model.

[0078] like Figure 3 As shown, the first branch 310 may include a preprocessing module 311 and a feature extraction module. The preprocessing module 311 may be used to preprocess the hand-drawn image to obtain an intermediate image corresponding to the hand-drawn image, and the intermediate image may be input into the feature extraction module. The feature extraction module may be used to extract features from the intermediate image to obtain hand-drawn features corresponding to the hand-drawn image, and the hand-drawn features may be input into the second branch 320.

[0079] In some embodiments, the preprocessing module 311 may be a learnable adapter module. The adapter module may perform dimensionality reduction processing on the hand-drawn image through a pixel unshuffle operation to reduce the dimension of the hand-drawn image to a preset dimension. That is, the intermediate image corresponding to the hand-drawn image may be an image of a preset dimension. It should be noted that the preset dimension may be determined according to the actual scene, and the embodiment of the present application does not limit this. For example, the preset dimension may be determined to be 64*64 according to the actual scene.

[0080] In some other embodiments, the feature extraction module may include a plurality of network stages connected in sequence. The feature extraction module may extract features through each network stage, and may input the hand-drawn features extracted by each network stage into the second branch 320 respectively.

[0081] For example, Figure 3As shown, each network layer may include one common convolution and two residual convolutions. That is, each network layer may perform feature extraction through one common convolution and two residual convolutions to obtain the hand-drawn features extracted by the network layer.

[0082] It should be noted that the number of network layers included in the feature extraction module can be determined according to the actual scenario, and the embodiment of the present application does not limit this. For example, it can be determined according to the actual scenario that the feature extraction module includes four network layers. Figure 3 As shown, the feature extraction module may include a first network layer 312, a second network layer 313, a third network layer 314 and a fourth network layer 315 connected in sequence.

[0083] In some embodiments, a downsampling layer may be included between any two network layers to perform feature downsampling processing through the downsampling layer. Figure 3 As shown, a first downsampling layer may be included between the first network layer 312 and the second network layer 313 , a second downsampling layer may be included between the second network layer 313 and the third network layer 314 , and a third downsampling layer may be included between the third network layer 314 and the fourth network layer 315 .

[0084] After obtaining the intermediate image corresponding to the hand-drawn image, the first network layer 312 of the feature extraction module can be used to extract features from the intermediate image to obtain the first hand-drawn feature (i.e. Figure 3 As shown in ), and the first hand-drawn feature can be input into the second branch 320.

[0085] After obtaining the first hand-drawn feature, the first hand-drawn feature can be downsampled by the first downsampling layer of the feature extraction module, and the downsampled first hand-drawn feature can be processed by the second network layer 313 to obtain the second hand-drawn feature corresponding to the hand-drawn image (i.e. Figure 3 As shown in ), the second hand-drawn feature can be input into the second branch 320.

[0086] After obtaining the second hand-drawn feature, the second hand-drawn feature can be downsampled by the second downsampling layer of the feature extraction module, and the downsampled second hand-drawn feature can be processed by the third network layer 314 to obtain the third hand-drawn feature corresponding to the hand-drawn image (i.e. Figure 3 As shown in ), the third hand-drawn feature can be input into the second branch 320.

[0087] After obtaining the third hand-drawn feature, the third hand-drawn feature can be downsampled by the third downsampling layer of the feature extraction module, and the downsampled third hand-drawn feature can be processed by the fourth network layer 315 to obtain the fourth hand-drawn feature corresponding to the hand-drawn image (i.e. Figure 3 As shown in ), the fourth hand-drawn feature can be input into the second branch 320.

[0088] It should be noted that the dimensions of the first hand-drawn feature, the second hand-drawn feature, the third hand-drawn feature, and the fourth hand-drawn feature can be determined according to the actual scene, and the embodiment of the present application does not limit this. For example, according to the actual scene, the dimension of the first hand-drawn feature can be determined to be 64*64, the dimension of the second hand-drawn feature can be determined to be 32*32, the dimension of the third hand-drawn feature can be determined to be 16*16, and the dimension of the fourth hand-drawn feature can be determined to be 8*8.

[0089] like Figure 3 As shown, the second branch 320 may include a text encoder 321, a denoising module 322, and a decoder 323. The text encoder 321 may be used to encode text content to obtain a text vector corresponding to the text content, and the text vector may be input into the denoising module 322. In one example, the text encoder 321 may be a contrastive language-image pre-training (CLIP) model based on contrastive text-image pairs.

[0090] The denoising module 322 can generate hidden features of an image related to the hand-drawn image in a latent space according to the text vector, the hand-drawn features input by the first branch 310, and the random noise, and can input the hidden features into the decoder 323. The decoder 323 can decode the hidden features to obtain a generated image related to the hand-drawn image.

[0091] In some embodiments, the denoising module 322 may be a U-Net network. The U-Net network may obtain the hidden features of the generated image in the latent space by iteratively denoising the random noise. In each denoising process, the U-Net network may predict the noise based on the guidance of the text content (i.e., text vector) and the hand-drawn image (i.e., hand-drawn features). After obtaining the predicted noise, the U-Net network may remove the predicted noise from the random noise, and may use the noise after removing the predicted noise as the new input of the U-Net network, and perform the next denoising process again, and so on, until the noise meets the preset conditions, or the number of iterations reaches the preset number, so as to obtain the hidden features of the generated image in the latent space. In this case, the preset conditions and the preset number of times may be specifically determined according to the actual scenario, and the embodiments of the present application are not limited thereto.

[0092] For example, Figure 3 As shown, the U-Net network can detect random noise Z T Processing to obtain random noise Z T The corresponding features can be used to compare the text vector and hand-drawn features with the random noise Z T The corresponding features are fused to predict the random noise Z T After getting the predicted noise, the U-Net network can T Subtract the predicted noise to get the noise Z T-1 , and the noise Z T-1 As the new input of the U-Net network, re-noise Z T-1 To perform noise reduction. That is, the U-Net network can reduce the noise Z T-1 After processing, we get the noise Z T-1 The corresponding features can be used to compare the text vector and hand-drawn features with the noise Z T-1 The corresponding features are fused to predict the noise Z T-1 After obtaining the predicted noise, the U-Net network can T-1 Subtract the predicted noise to get the noise Z --2 , and the noise Z T-2 As the new input of the U-Net network, re-noise Z T-2 Perform noise reduction processing, and so on, until the noise Z is obtained 0 , that is, the hidden features of the generated image in the latent space are obtained. 0 After that, the U-Net network can decode the noise Z 0 Decoding is performed to obtain a generated image.

[0093] Exemplarily, the U-Net network may include multiple downsampling modules and multiple upsampling modules. The U-Net network may first use the downsampling module to T Downsampling is performed to obtain random noise Z T The corresponding down-sampled features can be combined with the text vector and hand-drawn features and the random noise Z T The corresponding down-sampled features are fused to obtain the fused features. After obtaining the fused features, the U-Net network can upsample the fused features through the upsampling module to predict the random noise Z T The noise in the

[0094] For example, Figure 3As shown, the U-Net network may include four down-sampling modules and four up-sampling modules. That is, the U-Net network may include a first down-sampling module, a second down-sampling module, a third down-sampling module, and a fourth down-sampling module, and may include a first up-sampling module, a second up-sampling module, a third up-sampling module, and a fourth up-sampling module.

[0095] like Figure 3 As shown, the U-Net network can process the random noise Z through the first downsampling module T Downsampling is performed to obtain random noise Z T The first down-sampled feature corresponding to the text vector and the hand-drawn feature (such as the first hand-drawn feature) can be fused to obtain the first fused feature. After obtaining the first fused feature, the U-Net network can down-sample the first fused feature through the second down-sampling module to obtain the random noise Z T The corresponding second down-sampled feature can be fused with the text vector and the hand-drawn feature (such as the second hand-drawn feature) to obtain the second fused feature. After obtaining the second fused feature, the U-Net network can down-sample the second fused feature through the third down-sampling module to obtain the random noise Z T The corresponding third down-sampled feature can be fused with the text vector and the hand-drawn feature (such as the third hand-drawn feature) to obtain a third fused feature. After obtaining the third fused feature, the U-Net network can down-sample the third fused feature through the fourth down-sampling module to obtain the random noise Z T The corresponding fourth down-sampled feature can be fused with the text vector and the hand-drawn feature (eg, the fourth hand-drawn feature) to obtain a fourth fused feature.

[0096] After obtaining the fourth fusion feature, the U-Net network can upsample the fourth fusion feature through the first upsampling module to obtain the random noise Z T The first up-sampled feature corresponding to the text vector can be fused with the first up-sampled feature to obtain the fifth fused feature. After obtaining the fifth fused feature, the U-Net network can up-sample the fifth fused feature through the second up-sampling module to obtain the random noise Z T The corresponding second up-sampled feature can be fused with the text vector to obtain the sixth fused feature. After obtaining the sixth fused feature, the U-Net network can up-sample the sixth fused feature through the third up-sampling module to obtain the random noise Z TThe corresponding third up-sampled feature can be fused with the text vector to obtain the seventh fused feature. After obtaining the seventh fused feature, the U-Net network can up-sample the seventh fused feature through the fourth up-sampling module to obtain the random noise Z T The corresponding fourth up-sampled feature can be fused with the text vector to predict the random noise Z T The corresponding noise.

[0097] In order to obtain the random noise Z T After the corresponding noise, the U-Net network can T Subtract the predicted noise to get the noise Z T-1 , and the noise Z T-1 As the new input of the U-Net network, re-noise Z T-1 Perform noise reduction processing, and so on, until the noise Z is obtained 0 . After getting the noise Z 0 After that, the U-Net network can decode the noise Z 0 Decoding is performed to obtain a generated image.

[0098] In some embodiments, the image generation model may be an image generation model trained based on multiple sets of training data, wherein each set of training data may include a sample image, a training hand-drawn image drawn based on the sample image, and training text content corresponding to the sample image.

[0099] It should be understood that the loss function used by the image generation model during training can be:

[0100]

[0101] Among them, L simple is the loss function, t is the time, c t is the training text content corresponding to the sample image, F c To train the hand-drawn features corresponding to the hand-drawn images, x 0 is a sample image, x t is the generated image at time t, ε is the actual noise added, and ε θ (x t ,c t ,t,F c ) is the predicted noise at time t.

[0102] It should be noted that during the training process of the image generation model, the learning parameters of the second branch 320 can be frozen, so that the image generation model can be guided step by step to learn the hand-drawn features through the above-mentioned loss function, so that the hand-drawn features (such as the structural information of the hand-drawn image) can affect the process of the second branch 320, thereby allowing the hand-drawn image to guide the generation of the image.

[0103] In addition, the execution entity for training the image generation model can be an electronic device that generates images through the image generation model, or it can be other devices. The embodiment of the present application does not impose any specific restrictions on the execution entity for training the image generation model, and it can be determined according to the actual scenario.

[0104] In some embodiments, the image generation model may further include a third branch, which may be a low-rank adaptation (LoRA) model. The third branch may be arranged next to the U-Net network. The third branch may be used to fine-tune the image generation model according to a preset image style, so that the image generation model may output a generated image of a preset image style, increase the fun of the generated image, and satisfy the user experience.

[0105] It should be noted that the preset image style can be determined according to the actual scene, and the embodiment of the present application does not limit this. For example, the preset image style can be determined according to the actual scene to be one or more of isometric style, technological style, landscape style or cyberpunk style.

[0106] It should be understood that the LoRA model can learn features from a series of images, and can transfer the learned features such as specific artistic style or modeling to the generated image. The LoRA model can simulate the change in parameters through low-rank decomposition, thereby realizing indirect training of large models with a very small amount of parameters. Among them, the content of fine-tuning the image generation model through the LoRA model can refer to the related content of fine-tuning the model through the LoRA model in the prior art. For the sake of simplicity, it will not be repeated here.

[0107] The image generation method provided in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings and specific application scenarios.

[0108] See also Figure 4 , Figure 4 FIG. 1 is a schematic flow chart of an image generation method provided in an embodiment of the present application. The method can be applied to electronic devices. Figure 4 As shown, the method may include:

[0109] S401. The electronic device obtains a hand-drawn image and text content.

[0110] In the embodiment of the present application, when a user generates an image through AI painting, the user can draw a sketch of the image to be generated to obtain a hand-drawn image, and can provide a text description of the image to be generated. The electronic device can obtain the hand-drawn image drawn by the user and the text content input.

[0111] S402: The electronic device inputs the hand-drawn image and text content into a trained image generation model for processing to obtain a generated image output by the image generation model.

[0112] After obtaining the hand-drawn image drawn by the user and the input text content, the electronic device can input the hand-drawn image and text content into the trained image generation model. The image generation model can generate an image related to the hand-drawn image (i.e., generate an image) based on the guidance of the hand-drawn image and text content.

[0113] It should be noted that the process of generating an image related to the hand-drawn image by the electronic device based on the guidance of the hand-drawn image and the text content can refer to the aforementioned Figure 3 Related description in .

[0114] For example, Figure 3 As shown, the image generation model may include a first branch 310 and a second branch 320. The first branch 310 may be used to extract features from the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and the hand-drawn features may be input into the second branch 320. The second branch 320 may be used to generate a generated image related to the hand-drawn image based on the hand-drawn features and text content.

[0115] In some embodiments, the image generation model may be a model fine-tuned based on the LoRA model. After obtaining the hand-drawn image and text content, the image generation model may generate a generated image of a preset image style according to the hand-drawn image and text content.

[0116] In one example, the image generation model fine-tuned based on the LoRA model can be a model that supports multiple preset image styles. When a user wants to generate an image of a preset image style (e.g., preset image style A), the user can describe the preset image style A to be generated through text content. The image generation model can generate a generated image of the preset image style A based on the hand-drawn image and text content (including the preset image style A).

[0117] See also Figure 5 and Figure 6 , Figure 5 and Figure 6 A schematic diagram of an application scenario provided by an embodiment of the present application is shown.

[0118] like Figure 5As shown in (a), when the user wants to generate a chair image, the user can draw a sketch of the chair image, that is, a hand-drawn image 510 of the chair, and can provide a text description of the chair image to be generated. Assuming that the preset image style A that the user wants is an isometric style, the text content input by the user can be "comfortable chair, isometric style". After the electronic device obtains the hand-drawn image 510 and the text content (for example, "comfortable chair, isometric style"), the hand-drawn image 510 and the text content can be input into the image generation model. The image generation model can generate one or more chair images 511 in an isometric style based on the hand-drawn image 510 and the text content (for example, "comfortable chair, isometric style").

[0119] like Figure 5 As shown in (b), when the user wants to generate a car image, the user can draw a sketch of the car image, that is, a hand-drawn image 520 of the car, and can provide a text description of the image of the car to be generated. Assuming that the preset image style A that the user wants is a technology style, the text content input by the user can be "future car, technology style". After the electronic device obtains the hand-drawn image 520 and the text content (for example, "future car, technology style"), the hand-drawn image 520 and the text content can be input into the image generation model. The image generation model can generate one or more car images 521 in a technology style based on the hand-drawn image 520 and the text content (for example, "future car, technology style").

[0120] like Figure 5 As shown in (c) in the figure, when a user wants to generate a landscape image, the user can draw a sketch of the landscape image, i.e., a hand-painted image 530, and can provide a text description of the landscape image to be generated. Assuming that the preset image style A that the user wants is a landscape-painting style, the text content input by the user can be "green mountains, green trees, blue sky, white clouds, small rivers, scenery, landscape style". After the electronic device obtains the hand-painted image 530 and the text content (e.g., "green mountains, green trees, blue sky, white clouds, small rivers, scenery, landscape style"), the hand-painted image 530 and the text content can be input into the image generation model. The image generation model can generate one or more landscape images 531 in a landscape style based on the hand-painted image 530 and the text content (e.g., "green mountains, green trees, blue sky, white clouds, small rivers, scenery, landscape style").

[0121] like Figure 6As shown, when the user wants to generate a city image, the user can draw a sketch of the city image, that is, a hand-drawn image 610, and can provide a text description of the city image to be generated. Assuming that the preset image style A that the user wants is a cyberpunk style, the text content input by the user can be "future buildings, future cities, city scenery, skyscrapers, outdoors, sky, neon lights, science fiction, cyberpunk style". After the electronic device obtains the hand-drawn image 610 and the text content (for example, "future buildings, future cities, city scenery, skyscrapers, outdoors, sky, neon lights, science fiction, cyberpunk style"), the hand-drawn image 610 and the text content can be input into the image generation model. The image generation model can generate one or more city images 611 in a cyberpunk style based on the hand-drawn image 610 and the text content (for example, "future buildings, future cities, city scenery, skyscrapers, outdoors, sky, neon lights, science fiction, cyberpunk style").

[0122] It should be noted that the image generation model based on the fine-tuning of the LoRA model described above supports multiple preset image styles for exemplary explanation only and should not be understood as a limitation on the embodiments of the present application. In the embodiments of the present application, the image generation model based on the fine-tuning of the LoRA model may also support only one preset image style. When only one preset image style is supported, the image generation model can generate an image of the preset image style based on hand-drawn images and text content. At this time, the user does not need to describe the preset image style through text content. In other words, when the image generation model based on the fine-tuning of the LoRA model only supports one preset image style (for example, preset image style B), when the user does not enter the desired image style in the text content, the image generation model can generate a generated image of the preset image style B based on the hand-drawn image and text content (i.e., does not contain a description of the image style).

[0123] In some embodiments, in order to improve the effect of generating an image and improve the user experience, when the generated image is obtained based on the hand-drawn image and the text content through the image generation model, the electronic device can also obtain preset detail content, which can be content in the form of text, and can input the preset detail content into the image generation model. The image generation model can output the generated image based on the hand-drawn image, the text content and the detail content.

[0124] In one example, when an image generation model outputs a generated image based on a hand-drawn image, text content, and detail content, the image generation model can respectively perform vectorization processing (e.g., embedding) on ​​the hand-drawn image, text content, and detail content to obtain a vectorized representation corresponding to the hand-drawn image, a vectorized representation corresponding to the text content, and a vectorized representation corresponding to the detail content, so that the image generation model can efficiently and quickly output the generated image based on the vectorized representation corresponding to the hand-drawn image, the vectorized representation corresponding to the text content, and the vectorized representation corresponding to the detail content, and can improve the quality and accuracy of the generated image.

[0125] It should be noted that the preset details can be specifically determined according to the actual scene, and the embodiments of the present application are not limited to this. For example, the preset details can be determined to include "8k, ultra-high definition, ultra-realistic, ultra-detailed, bright, surrealism, ultra-fine photography" according to the actual scene.

[0126] In an embodiment of the present application, when performing AI painting, the user can draw a sketch corresponding to the desired image (i.e., a hand-drawn image), and can input text content (i.e., a text description corresponding to the desired image). After the electronic device obtains the hand-drawn image and text content, the hand-drawn image and text content can be input into a trained image generation model for processing to obtain a generated image output by the image generation model; wherein the image generation model includes a first branch and a second branch, the first branch is used to extract features from the hand-drawn image, obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch, the second branch is used to process the hand-drawn features and text content to obtain a generated image, and the second branch is a stable diffusion model. That is, in an embodiment of the present application, the image generation model can accurately generate the image desired by the user based on the hand-drawn image drawn by the user and the text content input, which can meet the actual needs of the user and improve the user experience.

[0127] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0128] Corresponding to the image generation method described in the above embodiment, Figure 7 A structural block diagram of an image generating device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0129] Reference Figure 7 , the device may include:

[0130] A hand-drawn image acquisition module 701 is used to acquire hand-drawn images and text content;

[0131] An image generation module 702 is used to input the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model;

[0132] Among them, the image generation model includes a first branch and a second branch. The first branch is used to extract features of the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch. The second branch is used to process the hand-drawn features and the text content to obtain the generated image. The second branch is a stable diffusion model.

[0133] In some embodiments, the first branch includes a preprocessing module and a feature extraction module, and the image generation module 702 is specifically used to input the hand-drawn image and the text content into the image generation model; perform dimensionality reduction processing on the hand-drawn image through the preprocessing module in the first branch to obtain an intermediate image corresponding to the hand-drawn image; perform feature extraction on the intermediate image through the feature extraction module in the first branch to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch; process the hand-drawn features and the text content through the second branch to obtain the generated image output by the image generation model.

[0134] In a possible implementation, the feature extraction module includes a first network layer, a first downsampling layer, a second network layer, a second downsampling layer, a third network layer, a third downsampling layer and a fourth network layer.

[0135] The image generation module 702 is further used to extract features from the intermediate image through the first network layer to obtain a first hand-drawn feature corresponding to the hand-drawn image, and input the first hand-drawn feature into the second branch; downsample the first hand-drawn feature through the first downsampling layer, and process the downsampled first hand-drawn feature through the second network layer to obtain a second hand-drawn feature corresponding to the hand-drawn image, and input the second hand-drawn feature into the second branch; downsample the second hand-drawn feature through the second downsampling layer, and process the downsampled second hand-drawn feature through the third network layer to obtain a third hand-drawn feature corresponding to the hand-drawn image, and input the third hand-drawn feature into the second branch; downsample the third hand-drawn feature through the third downsampling layer, and process the downsampled third hand-drawn feature through the fourth network layer to obtain a fourth hand-drawn feature corresponding to the hand-drawn image, and input the fourth hand-drawn feature into the second branch.

[0136] In other embodiments, the image generation model also includes a third branch, which is a low-rank adaptive LoRA model; the third branch is used to fine-tune the image generation model according to a preset image style, so that the image generation model outputs a generated image of the preset image style.

[0137] In some embodiments, the apparatus may further include:

[0138] A detail content acquisition module is used to acquire preset detail content, where the detail content is in text form;

[0139] The image generation module is further used to input the hand-drawn image, the text content and the detail content into the image generation model for processing to obtain the generated image output by the image generation model.

[0140] In some embodiments, the apparatus may further include:

[0141] A training module is used to train the image generation model according to multiple groups of training data to obtain the trained image generation model, wherein each group of training data includes a sample image, a training hand-drawn image drawn based on the sample image, and training text content corresponding to the sample image.

[0142] In one example, the loss function used by the image generation model during training is:

[0143]

[0144] Among them, L simple is the loss function, t is the time, c t is the training text content corresponding to the sample image, F c is the hand-drawn feature corresponding to the training hand-drawn image, x 0 is the sample image, x t is the generated image at time t, ε is the real noise, ε θ (x t ,c t ,t,F c ) is the predicted noise at time t.

[0145] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0146] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0147] Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown), a memory 81, and a computer program 82 stored in the memory 81 and executable on the at least one processor 80, wherein the processor 80 implements the steps of any of the above-mentioned image generation method embodiments when executing the computer program 82.

[0148] The electronic device 8 may be a computing device such as a desktop computer, a notebook, a PDA, a wearable device, a vehicle-mounted device, a cloud server, etc. The electronic device 8 may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will appreciate that Figure 8 It is only an example of the electronic device 8 and does not constitute a limitation on the electronic device 8. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0149] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0150] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, for example, a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may also be an external storage device of the electronic device 8, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 8. Further, the memory 81 may also include both an internal storage unit and an external storage device of the electronic device 8. The memory 81 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0151] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0152] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable storage medium cannot be an electric carrier signal and a telecommunication signal.

[0154] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0155] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0156] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0157] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: include: Get hand-drawn images and text content; Inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model; Among them, the image generation model includes a first branch and a second branch. The first branch is used to extract features of the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch. The second branch is used to process the hand-drawn features and the text content to obtain the generated image. The second branch is a stable diffusion model.

2. The method according to claim 1, characterized in that The first branch includes a preprocessing module and a feature extraction module, and the hand-drawn image and the text content are input into a trained image generation model for processing to obtain a generated image output by the image generation model, including: Inputting the hand-drawn image and the text content into the image generation model; Performing dimensionality reduction processing on the hand-drawn image through the preprocessing module in the first branch to obtain an intermediate image corresponding to the hand-drawn image; Performing feature extraction on the intermediate image by using the feature extraction module in the first branch to obtain hand-drawn features corresponding to the hand-drawn image, and inputting the hand-drawn features into the second branch; The hand-drawn features and the text content are processed through the second branch to obtain the generated image output by the image generation model.

3. The method according to claim 2, characterized in that The feature extraction module includes a first network layer, a first downsampling layer, a second network layer, a second downsampling layer, a third network layer, a third downsampling layer and a fourth network layer. The feature extraction module in the first branch extracts features from the intermediate image to obtain hand-drawn features corresponding to the hand-drawn image, and inputs the hand-drawn features into the second branch, including: Performing feature extraction on the intermediate image through the first network layer to obtain a first hand-drawn feature corresponding to the hand-drawn image, and inputting the first hand-drawn feature into the second branch; Downsampling the first hand-drawn feature through the first downsampling layer, processing the downsampled first hand-drawn feature through the second network layer to obtain a second hand-drawn feature corresponding to the hand-drawn image, and inputting the second hand-drawn feature into the second branch; Downsampling the second hand-drawn feature through the second downsampling layer, processing the downsampled second hand-drawn feature through the third network layer to obtain a third hand-drawn feature corresponding to the hand-drawn image, and inputting the third hand-drawn feature into the second branch; The third hand-drawn feature is downsampled by the third downsampling layer, and the downsampled third hand-drawn feature is processed by the fourth network layer to obtain a fourth hand-drawn feature corresponding to the hand-drawn image, and the fourth hand-drawn feature is input into the second branch.

4. The method according to claim 1, characterized in that: The image generation model also includes a third branch, which is a low-rank adaptive LoRA model; The third branch is used to fine-tune the image generation model according to a preset image style, so that the image generation model outputs a generated image of the preset image style.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining preset detailed content, wherein the detailed content is content in text form; The step of inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model includes: The hand-drawn image, the text content and the detail content are input into the image generation model for processing to obtain the generated image output by the image generation model.

6. The method according to any one of claims 1 to 5, characterized in that Before inputting the hand-drawn image and the text content into the trained image generation model for processing, the method further includes: The image generation model is trained according to multiple groups of training data to obtain the trained image generation model, wherein each group of training data includes a sample image, a training hand-drawn image drawn based on the sample image, and training text content corresponding to the sample image.

7. The method according to claim 6, characterized in that The loss function used by the image generation model during training is: Among them, L simple is the loss function, t is the time, c t is the training text content corresponding to the sample image, F c is the hand-drawn feature corresponding to the training hand-drawn image, x0 is the sample image, x t is the generated image at time t, ε is the actual noise added, and ε θ (x t ,c t ,t,F c ) is the predicted noise at time t.

8. An image generating device, characterized in that: include: A hand-drawn image acquisition module is used to acquire hand-drawn images and text content; An image generation module, used for inputting the hand-drawn image and the text content into a trained image generation model for processing to obtain a generated image output by the image generation model; Among them, the image generation model includes a first branch and a second branch. The first branch is used to extract features of the hand-drawn image to obtain hand-drawn features corresponding to the hand-drawn image, and input the hand-drawn features into the second branch. The second branch is used to process the hand-drawn features and the text content to obtain the generated image. The second branch is a stable diffusion model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the image generating method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a computer, the computer implements the image generating method according to any one of claims 1 to 7.

Images