Image generation model training method and device, image generation method and device, equipment and medium

Abstract

The invention relates to an image generation model training method and device, an image generation method and device, equipment and a medium. The image generation model training method comprises the steps that a training sample set is acquired, and each training sample in the training sample set comprises an image sketch, a text and a real image; for each training sample, inputting the real image and the text into a pre-training diffusion model, and generating a prediction image based on the real image and the text by the pre-training diffusion model; performing fine tuning training on the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model; and performing fine tuning training on the first diffusion model based on the image sketch to obtain a second diffusion model, and determining the second diffusion model as a trained diffusion model, so that the pre-trained diffusion model can be trained through the image sketch and the text, and the quality and accuracy of the image generated by the trained diffusion model are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image generation technology, and in particular to an image generation model training, an image generation method, a device, a equipment and a medium. Background Art

[0002] With the continuous development of image technology, image generation technology has also become an important branch in the field of computer vision. Currently, most image generation technologies are based on generative adversarial networks. This method encodes the user's text description of the image as conditional information and inputs it into the generator of the generative adversarial network, thereby guiding the generator based on the conditional information to generate an image corresponding to the conditional information.

[0003] However, due to the limitations of the generative adversarial network structure, there are problems with the generated images having low resolution and data overfitting, resulting in low quality and accuracy of the generated images. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an image generation model training, an image generation method, an apparatus, a device and a medium.

[0005] A first aspect of an embodiment of the present disclosure provides an image generation model training method, comprising:

[0006] A training sample set is obtained, where each training sample in the training sample set includes an image sketch, text, and a real image, and the text is used to describe information of the real image;

[0007] For each training sample, the real image and text are input into the pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and text; the pre-trained diffusion model is fine-tuned based on the predicted image and the real image to obtain a first diffusion model;

[0008] The first diffusion model is fine-tuned and trained based on the image sketch to obtain a second diffusion model, and the second diffusion model is determined as the trained diffusion model.

[0009] A second aspect of the embodiments of the present disclosure provides an image generation method, including:

[0010] Obtaining a sketch to be processed and text to be processed, wherein the text to be processed is used to describe information of an image to be generated;

[0011] The sketch to be processed and the text to be processed are input into the trained diffusion model, and the trained diffusion model generates a target image based on the sketch to be processed and the text to be processed. The trained diffusion model is trained using the image generation model training method described in the first aspect.

[0012] A third aspect of the embodiments of the present disclosure provides an image generation model training device, including:

[0013] A training set acquisition unit, used for acquiring a training sample set, each training sample in the training sample set includes an image sketch, text and a real image, and the text is used for describing information of the real image;

[0014] A first training unit is used to input the real image and text into the pre-trained diffusion model for each training sample, so that the pre-trained diffusion model generates a predicted image based on the real image and text; and fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model;

[0015] The second training unit is used to fine-tune the first diffusion model based on the image sketch to obtain a second diffusion model, and determine the second diffusion model as the trained diffusion model.

[0016] A fourth aspect of the embodiments of the present disclosure provides an image generating device, including:

[0017] A data acquisition unit, used to acquire a sketch to be processed and a text to be processed, wherein the text to be processed is used to describe information of an image to be generated;

[0018] An image generation unit is used to input the sketch to be processed and the text to be processed into a trained diffusion model, and the trained diffusion model generates a target image based on the sketch to be processed and the text to be processed. The trained diffusion model is trained using the image generation model training method described in the first aspect above.

[0019] A fifth aspect of the embodiments of the present disclosure provides an electronic device, including:

[0020] processor;

[0021] A memory for storing executable instructions;

[0022] Among them, the processor is used to read executable instructions from the memory and execute the executable instructions to implement the image generation model training method provided by the first aspect or the image generation method provided by the second aspect.

[0023] The sixth aspect of an embodiment of the present disclosure provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the image generation model training method provided by the first aspect or the image generation method provided by the second aspect.

[0024] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0025] The image generation model training, image generation method, device, equipment and medium provided by the embodiments of the present disclosure can obtain a training sample set, each training sample in the training sample set includes an image sketch, text and a real image, and the text is used to describe the information of the real image. After obtaining the training sample set, for each training sample, the real image and text are input into the pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and text, and then the pre-trained diffusion model is fine-tuned based on the predicted image and the real image to obtain a first diffusion model, so that the first diffusion model has the ability to generate images with the same style as the training sample set. Then, the first diffusion model is fine-tuned based on the image sketch to obtain a second diffusion model, and the second diffusion model is determined as the diffusion model that has been trained, so that the diffusion model that has been trained can generate images based on the input multimodal features, namely the features of the image sketch and the text features, thereby improving the quality and accuracy of the generated image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0028] Figure 1 is a flowchart of an image generation model training method provided by an embodiment of the present disclosure;

[0029] Figure 2 is a structural schematic diagram of an adapter provided by an embodiment of the present disclosure;

[0030] Figure 3 is a structural diagram of a residual module provided by an embodiment of the present disclosure;

[0031] Figure 4 is a flowchart of another image generation model training method provided by an embodiment of the present disclosure;

[0032] Figure 5 is a flow chart of an image generating method provided by an embodiment of the present disclosure;

[0033] Figure 6 It is a structural schematic diagram of an image generation model training device provided by an embodiment of the present disclosure;

[0034] Figure 7is a structural schematic diagram of an image generating device provided by an embodiment of the present disclosure;

[0035] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0037] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0038] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0039] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0041] Generally speaking, most of the current image generation technologies are based on generative adversarial networks. This method encodes the user's text description of the image as conditional information and inputs it into the generator of the generative adversarial network, thereby guiding the generator based on the conditional information to generate an image corresponding to the conditional information.

[0042] However, due to the limitations of the generative adversarial network structure, there are problems such as low resolution of the generated images and overfitting of the data, resulting in low quality and accuracy of the generated images. To address this problem, the present disclosure provides an image generation model training method, which is described below in conjunction with a specific embodiment.

[0043] Figure 1 It is a flowchart of an image generation model training method provided by an embodiment of the present disclosure. The method can be executed by an image generation model training device. The image generation model training device can be implemented in software and / or hardware. The image generation model training device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc.

[0044] like Figure 1 As shown, the image generation model training method provided in this embodiment includes the following steps.

[0045] S110, obtaining a training sample set, where each training sample in the training sample set includes an image sketch, text, and a real image, and the text is used to describe information about the real image.

[0046] In the disclosed embodiments, the image sketch may be an image that is simplified, abstracted, and quickly sketched out of an image or scene by hand-drawing or using a drawing tool. It focuses more on information such as the form, structure, and spatial relationship of objects in the image. The image sketch is a sketch corresponding to a real image.

[0047] The text is the text describing the information of the real image, especially the detail information. In the embodiment of the present disclosure, the text can be understood as the text supplementing the detail information of the real image not included in the image sketch.

[0048] In the disclosed embodiment, the styles of real images are different, the styles of images corresponding to the training sample sets are also different, and thus the styles of images generated by the trained diffusion model are also different. For example, the style of the image can be animation, oil painting, watercolor, etc. For example, the image sketch can be an animation sketch, etc.

[0049] In some embodiments of the present disclosure, the electronic device may parse the model training instruction in response to the user's model training instruction, and obtain a training sample set from a preset database based on the parsing result.

[0050] In other embodiments of the present disclosure, the electronic device may respond to a user's model training instruction, parse the model training instruction, determine style information corresponding to the training sample set based on the parsing result, and obtain the training sample set from a preset website by crawling based on the style information.

[0051] S120. For each training sample, input the real image and text into the pre-trained diffusion model, and generate a predicted image based on the real image and text by the pre-trained diffusion model; fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model.

[0052] In the embodiments of the present disclosure, the pre-trained diffusion model can be understood as a diffusion model obtained after pre-training. The pre-trained model has learned some basic characteristics or distribution laws of the data, thereby speeding up the efficiency and generation quality of subsequent progressive model training.

[0053] Exemplarily, the pre-trained diffusion model may be a stable diffusion model.

[0054] Among them, fine-tuning training can be understood as mainly adjusting the weights of the cross-attention layer of the pre-trained diffusion model that interacts with text information. In this way, the diverse generation capabilities of the pre-trained diffusion model are retained, the generation quality of the model in the field of image generation is improved, the number of parameters for model training is reduced, and the training rate of the model is accelerated.

[0055] Specifically, after acquiring the training sample set, the electronic device inputs the real image and text in the training sample into the pre-trained diffusion model for each training sample, and the pre-trained diffusion model extracts features of the real image and text respectively, and generates a predicted image based on the extracted feature vector. Then, after obtaining the predicted image, a first loss value is determined based on the predicted image and the real image and a preset loss function, and the parameters of the pre-trained model are adjusted based on the first loss value to obtain a first diffusion model, so that the obtained first diffusion model can have the ability to generate images with the same image style as the training sample set.

[0056] S130 , fine-tune the first diffusion model based on the image sketch to obtain a second diffusion model, and determine the second diffusion model as the trained diffusion model.

[0057] Specifically, after obtaining the first diffusion model, the electronic device extracts features from the image sketch to obtain a target image feature vector corresponding to the image sketch, inputs the target image feature vector and a text feature vector corresponding to the text into the first diffusion model, fine-tunes the first diffusion model to obtain a second diffusion model, and determines the second diffusion model as the trained diffusion model, so that the trained diffusion model has the ability to generate images through image sketches and text.

[0058] In the disclosed embodiment, a training sample set can be obtained, each training sample in the training sample set includes an image sketch, text and a real image, and the text is used to describe the information of the real image. After the training sample set is obtained, for each training sample, the real image and text are input into the pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and text, and then the pre-trained diffusion model is fine-tuned based on the predicted image and the real image to obtain a first diffusion model, so that the first diffusion model has the ability to generate images with the same style as the training sample set. Then, the first diffusion model is fine-tuned based on the image sketch to obtain a second diffusion model, and the second diffusion model is determined as the diffusion model that has been trained, so that the diffusion model that has been trained can generate images based on the input multimodal features, namely the features of the image sketch and the text features, thereby improving the quality and accuracy of the generated images.

[0059] In an embodiment of the present disclosure, a real image and text are input into a pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and text. Specifically, the method may include: inputting the text into a text encoder of the pre-trained diffusion model, and the text encoder encodes the text to obtain a text feature vector; inputting the real image into an autoencoder of the pre-trained diffusion model, and the autoencoder encodes the real image to obtain a first image feature vector; inputting the first image feature vector into a noisy neural network, and the noisy neural network performs a noise process on the first image feature vector to obtain a second image feature vector; concatenating the second image feature vector and the text feature vector to obtain a first fused feature vector, inputting the first fused feature vector into a denoising neural network, and the denoising neural network performs a denoising process on the first fused feature vector, and decoding the target fused feature vector obtained after the denoising process based on the decoder to generate a predicted image.

[0060] Among them, the text encoder can be a contrastive language-image pre-training (Clip) encoder; the denoising neural network can be a UNet denoising neural network.

[0061] The specific implementation of performing noise addition processing on the first image feature vector and denoising processing on the first fused feature vector is similar to the existing implementation of performing noise addition processing and denoising processing on the image feature vector, and will not be described in detail here.

[0062] In the disclosed embodiment, it is possible to perform noise processing on the first image feature vector corresponding to the real image, and then generate a predicted image based on the first image feature vector obtained after the noise processing and the text feature vector corresponding to the text, and fine-tune the pre-trained diffusion model based on the predicted image, so that the obtained first diffusion model has the ability to generate an image with the same style as the real image based on the text, thereby improving the accuracy of image generation.

[0063] In an embodiment of the present disclosure, a first diffusion model is fine-tuned and trained based on an image sketch to obtain a second diffusion model, which may specifically include: inputting the image sketch into a preset adapter, encoding the image sketch based on the preset adapter, and obtaining a target image feature vector corresponding to the image sketch; performing vector concatenation processing on the target image feature vector and a text feature vector corresponding to the text to obtain a second fused feature vector; inputting the second fused feature vector into the first diffusion model, and fine-tuning the first diffusion model to obtain a second diffusion model.

[0064] Figure 2 is a schematic diagram of the structure of an adapter provided by an embodiment of the present disclosure, such as Figure 2 As shown, the preset adapter includes two branches, namely a first branch and a second branch. The first branch consists of a first convolutional layer and three first encoders, and the second branch consists of a second convolutional layer and three second encoders; wherein the first encoder includes three residual modules and a pooling layer, and the second encoder includes three residual modules, a third convolutional layer and a fourth convolutional layer.

[0065] The convolution kernel of the first convolution layer is larger than the convolution kernel of the second convolution layer. Exemplarily, the first convolution layer is a 5×5 convolution, and the second convolution layer is a 3×3 convolution.

[0066] The convolution kernel of the third convolution layer is larger than the convolution kernel of the fourth convolution layer. Exemplarily, the third convolution layer is a 3×3 convolution, and the fourth convolution layer is a 1×1 convolution.

[0067] In the disclosed embodiment, the preset adapter is a lightweight adapter. The preset adapter adopts a dual-branch structure to allow the model to capture features of different scales at the same time, thereby enhancing the model's ability to capture details and contextual information. The first branch captures a wider range of contextual information through larger convolution kernels, while the second branch focuses on finer details through smaller convolution kernels and residual modules, increasing the nonlinearity of the network and enabling the network to learn more complex features. The fourth convolution layer in the second branch increases the nonlinearity of the network by adopting 1×1 convolution, which helps the model learn more complex feature representations. This design can make the model more flexible and effective in handling complex visual tasks.

[0068] Furthermore, the residual modules in the first branch and the second branch adopt the same structure. Figure 3 is a schematic diagram of the structure of a residual module provided by an embodiment of the present disclosure, such as Figure 3 As shown in FIG. 1 , the residual module can be composed of an input layer, two convolutional layers (i.e., two-dimensional convolutional layer 1 and two-dimensional convolutional layer 2), an activation function, and an output layer. Among them, ⊕ represents the concatenation of the image feature vector input through the input layer and the image feature vector output by the two-dimensional convolutional layer 2 to obtain the image feature vector output by the output layer.

[0069] Specifically, after obtaining the first diffusion model, the electronic device inputs the image sketch into a preset adapter, and the preset adapter encodes the image sketch to obtain a target image feature vector corresponding to the image sketch. Then, the target image feature vector and the text feature vector corresponding to the text are vector-concatenated to obtain a second fused feature vector, which is input into the first diffusion model, and the first diffusion model generates an image based on the second fused feature vector, determines a second loss value based on the generated image, the real image, and a preset loss function, and adjusts the parameters of the first diffusion model based on the second loss value to obtain a second diffusion model.

[0070] In an embodiment of the present disclosure, before inputting the image sketch into a preset adapter and encoding the image sketch based on the preset adapter to obtain a target image feature vector corresponding to the image sketch, the image generation model training method may further include: performing pixel rearrangement processing on the image sketch to obtain a target sketch.

[0071] Among them, the pixel rearrangement processing of the image sketch can be understood as the pixel downsampling processing of the image sketch, thereby improving the rate of image sketch feature extraction and reducing the number of parameters and computational complexity of the model.

[0072] Furthermore, the image sketch is input into a preset adapter, and the image sketch is encoded based on the preset adapter to obtain a target image feature vector corresponding to the image sketch. Specifically, it may include: inputting the target sketch into the first branch of the preset adapter, encoding the target sketch by the first branch to obtain a third image feature vector, and inputting the target sketch into the second branch of the preset adapter, encoding the target sketch by the second branch to obtain a fourth image feature vector; the third image feature vector and the fourth image feature vector are concatenated to obtain a target image feature vector.

[0073] Specifically, the electronic device inputs the target sketch into the first branch and the second branch respectively, and the first convolution layer in the first branch performs convolution processing on the target sketch to obtain a first convolution vector, and the first convolution vector is input into the first encoder, and the first residual module in the first encoder processes the first convolution vector to obtain a second convolution vector, and the second convolution vector is input into the second residual module in the first encoder, and the second residual module processes the second convolution vector to obtain a third convolution vector, and the third convolution vector is input into the third residual module to obtain a fourth convolution vector, and the pooling layer performs pooling processing on the fourth convolution vector to obtain a third image feature vector; at the same time, the second convolution layer in the second branch performs convolution processing on the target sketch to obtain a fifth convolution vector, and the fifth convolution vector is input into the second encoder, and convolution processing is performed by three residual modules, the third convolution layer and the fourth convolution layer in sequence to obtain a fourth image feature vector; then, the third image feature vector and the fourth image feature vector are spliced ​​to obtain a target image feature vector.

[0074] The first residual module in the first encoder processes the first convolution vector to obtain the second convolution vector, which may specifically include: inputting the first convolution vector through the input layer; and performing the processing of the first convolution vector by the first two-dimensional convolution layer (such as Figure 3 The two-dimensional convolution layer 1 shown in FIG. 1 performs convolution processing on the first convolution vector to obtain a first result, and the first result is processed by an activation function to obtain a second result; the second result is input to the second two-dimensional convolution layer (as shown in FIG. Figure 3 The two-dimensional convolution layer 2) shown in FIG. 2 performs convolution processing on the second result to obtain a third result; concatenates the first convolution vector and the third result to obtain a second convolution vector, and then the output layer outputs the second convolution vector.

[0075] In the disclosed embodiment, image features are extracted from an image sketch through two branches of a preset adapter, which can capture contextual information of a wider range of image features, while focusing on detailed features in the image, increasing the nonlinearity of the network, and improving the richness and accuracy of the obtained image feature vector, thereby improving the accuracy of image generation by the trained diffusion model.

[0076] Figure 4 is a flowchart of another image generation model training method provided by an embodiment of the present disclosure. Figure 4 As shown, the image generation model training method may specifically include the following steps:

[0077] S410: Obtain a training sample set, where each training sample in the training sample set includes an image sketch, text, and a real image, and the text is used to describe information about the real image.

[0078] S420. For each training sample, the text is input into the text encoder of the pre-trained diffusion model, and the text encoder encodes the text to obtain a text feature vector; and the real image is input into the autoencoder of the pre-trained diffusion model, and the autoencoder encodes the real image to obtain a first image feature vector.

[0079] S430: Input the first image feature vector into a noise adding neural network, and use the noise adding neural network to perform noise adding processing on the first image feature vector to obtain a second image feature vector.

[0080] S440, concatenating the second image feature vector and the text feature vector to obtain a first fused feature vector, inputting the first fused feature vector into a denoising neural network, performing denoising on the first fused feature vector by the denoising neural network, and decoding the target fused feature vector obtained after the denoising based on the decoder to generate a predicted image.

[0081] S450 , fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model.

[0082] S460: Input the image sketch into a preset adapter, encode the image sketch based on the preset adapter, and obtain a target image feature vector corresponding to the image sketch.

[0083] S470 , performing vector concatenation processing on the target image feature vector and the text feature vector corresponding to the text to obtain a second fused feature vector.

[0084] S480: Input the second fused feature vector into the first diffusion model, perform fine-tuning training on the first diffusion model, and obtain a second diffusion model.

[0085] It should be noted that the specific implementation of steps S410-S480 is similar to the implementation of the relevant steps in the above-mentioned embodiment of the present disclosure, and will not be repeated here.

[0086] In the disclosed embodiment, the pre-trained diffusion model can be fine-tuned twice, so that the trained diffusion model not only has the ability to generate images with the same style as the user's requirements, but also can generate images based on a preset adapter combined with image sketches and text, thereby improving the accuracy of the generated images.

[0087] Figure 5 It is a flowchart of an image generating method provided by an embodiment of the present disclosure. The method can be executed by an image generating device. The image generating device can be implemented in software and / or hardware. The image generating device can be configured in an electronic device, such as a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc.

[0088] like Figure 5 As shown, the image generation method provided in this embodiment includes the following steps.

[0089] S510: Obtain a sketch to be processed and text to be processed, wherein the text to be processed is used to describe information of an image to be generated.

[0090] In some embodiments of the present disclosure, an electronic device can receive an image generation instruction in real time, wherein the image generation instruction includes a sketch to be processed and text to be processed required for generating an image. After receiving the image generation instruction, the image generation instruction is parsed to obtain the sketch to be processed and text to be processed.

[0091] In some other embodiments of the present disclosure, an electronic device can receive an image generation instruction in real time, wherein the image generation instruction includes identification information of a sketch to be processed and a text to be processed required for generating an image; after receiving the image generation instruction, the image generation instruction is parsed to obtain the identification information; and based on the identification information, the sketch to be processed and the text to be processed are obtained from a preset database.

[0092] S520: Input the sketch to be processed and the text to be processed into the trained diffusion model, and generate a target image based on the sketch to be processed and the text to be processed by the trained diffusion model.

[0093] In the embodiment of the present disclosure, the trained diffusion model is obtained by training using the image generation model training method described in the above embodiment of the present disclosure.

[0094] Specifically, after obtaining the sketch to be processed and the text to be processed, the electronic device inputs the sketch to be processed and the text to be processed into the trained diffusion model, and the trained diffusion model performs feature extraction on the sketch to be processed and the text to be processed to obtain the image feature vector corresponding to the sketch to be processed and the text feature vector corresponding to the text to be processed, and generates a target image based on the image feature vector corresponding to the sketch to be processed and the text feature vector corresponding to the text to be processed.

[0095] In the disclosed embodiment, the acquired sketch to be processed and the text to be processed can be input into a trained diffusion model, and the trained diffusion model can generate a target image based on the two modal features corresponding to the sketch and the text, thereby improving the quality and accuracy of the generated image.

[0096] Figure 6 It is a structural schematic diagram of an image generation model training device provided in an embodiment of the present disclosure.

[0097] In the disclosed embodiment, the image generation model training device can be set in an electronic device, which is understood as a part of the functional modules in the above electronic device. Specifically, the electronic device can be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.

[0098] like Figure 6 As shown, the image generation model training device 600 may include a training set acquisition unit 610, a first training unit 620 and a second training unit 630.

[0099] The training set acquisition unit 610 can be used to acquire a training sample set. Each training sample in the training sample set includes an image sketch, text, and a real image. The text is used to describe information about the real image.

[0100] The first training unit 620 can be used to input the real image and text into the pre-trained diffusion model for each training sample, and the pre-trained diffusion model generates a predicted image based on the real image and text; fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model.

[0101] The second training unit 630 may be configured to perform fine-tuning training on the first diffusion model based on the image sketch to obtain a second diffusion model, and determine the second diffusion model as a trained diffusion model.

[0102] In the disclosed embodiment, a training sample set can be obtained, each training sample in the training sample set includes an image sketch, text and a real image, and the text is used to describe the information of the real image. After the training sample set is obtained, for each training sample, the real image and text are input into the pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and text, and then the pre-trained diffusion model is fine-tuned based on the predicted image and the real image to obtain a first diffusion model, so that the first diffusion model has the ability to generate images with the same style as the training sample set. Then, the first diffusion model is fine-tuned based on the image sketch to obtain a second diffusion model, and the second diffusion model is determined as the diffusion model that has been trained, so that the diffusion model that has been trained can generate images based on the input multimodal features, namely the features of the image sketch and the text features, thereby improving the quality and accuracy of the generated images.

[0103] In some embodiments of the present disclosure, the first training unit 620 may be specifically used to input the text into a text encoder of the pre-trained diffusion model, and the text encoder encodes the text to obtain a text feature vector;

[0104] Inputting the real image into the autoencoder of the pre-trained diffusion model, the autoencoder encodes the real image to obtain a first image feature vector;

[0105] Inputting the first image feature vector into a noise adding neural network, and performing noise adding processing on the first image feature vector by the noise adding neural network to obtain a second image feature vector;

[0106] The second image feature vector and the text feature vector are concatenated to obtain a first fused feature vector, the first fused feature vector is input into a denoising neural network, the denoising neural network denoises the first fused feature vector, and the decoder is used to decode the target fused feature vector obtained after the denoising process to generate a predicted image.

[0107] In some embodiments of the present disclosure, the second training unit 630 may be specifically configured to input the image sketch into a preset adapter, perform encoding processing on the image sketch based on the preset adapter, and obtain a target image feature vector corresponding to the image sketch;

[0108] Performing vector concatenation processing on the target image feature vector and the text feature vector corresponding to the text to obtain a second fused feature vector;

[0109] The second fused feature vector is input into the first diffusion model, and the first diffusion model is fine-tuned and trained to obtain a second diffusion model.

[0110] In some embodiments of the present disclosure, the image generation model training device 600 may further include a pixel rearrangement unit.

[0111] The pixel rearrangement unit can be used to input the image sketch into a preset adapter, encode the image sketch based on the preset adapter, and obtain the target image feature vector corresponding to the image sketch before performing pixel rearrangement processing on the image sketch to obtain a target sketch.

[0112] In some embodiments of the present disclosure, the second training unit 630 may also be specifically configured to input the target sketch into a first branch of a preset adapter, and the first branch encodes the target sketch to obtain a third image feature vector, and input the target sketch into a second branch of a preset adapter, and the second branch encodes the target sketch to obtain a fourth image feature vector;

[0113] The third image feature vector and the fourth image feature vector are concatenated to obtain a target image feature vector.

[0114] In some embodiments of the present disclosure, the first branch consists of a first convolutional layer and three first encoders, the second branch consists of a second convolutional layer and three second encoders, and the convolution kernel of the first convolutional layer is larger than the convolution kernel of the second convolutional layer; wherein the first encoder includes three residual modules and a pooling layer, and the second encoder includes three residual modules, a third convolutional layer and a fourth convolutional layer.

[0115] It should be noted that Figure 6 The image generation model training device 600 shown can execute each step in the above method embodiment and realize each process and effect in the above method embodiment, which will not be described in detail here.

[0116] Figure 7 It is a structural schematic diagram of an image generating device provided by an embodiment of the present disclosure.

[0117] In the embodiment of the present disclosure, the image generating device may be arranged in an electronic device, and is understood as a part of the functional modules in the above electronic device. Specifically, the electronic device may be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.

[0118] like Figure 7 As shown, the image generating device 700 may include a data acquiring unit 710 and an image generating unit 720 .

[0119] The data acquisition unit 710 can be used to acquire a sketch to be processed and text to be processed, wherein the text to be processed is used to describe information of an image to be generated.

[0120] The image generation unit 720 can be used to input the sketch to be processed and the text to be processed into the trained diffusion model, and the trained diffusion model generates a target image based on the sketch to be processed and the text to be processed. The trained diffusion model is trained using the image generation model training method described in the first aspect above.

[0121] In the disclosed embodiment, the acquired sketch to be processed and the text to be processed can be input into a trained diffusion model, and the trained diffusion model can generate a target image based on the two modal features corresponding to the sketch and the text, thereby improving the quality and accuracy of the generated image.

[0122] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure.

[0123] In the disclosed embodiment, Figure 8 The electronic device shown may be a server or a terminal, wherein the terminal specifically includes a mobile phone, a computer or a tablet computer, etc., which is not limited here.

[0124] like Figure 8 As shown, the electronic device may include a processor 810 and a memory 820 storing computer program instructions.

[0125] Specifically, the processor 810 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0126] The memory 820 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 820 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 820 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 820 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 820 is a non-volatile solid-state memory. In a particular embodiment, the memory 820 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0127] The processor 810 reads and executes the computer program instructions stored in the memory 820 to perform the steps of the image generation model training method provided in the embodiment of the present disclosure.

[0128] In one example, the electronic device may further include a transceiver 830 and a bus 840. Figure 8 As shown, the processor 810, the memory 820 and the transceiver 830 are connected via a bus 840 and communicate with each other.

[0129] The bus 840 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 840 may include one or more buses.

[0130] The embodiments of the present disclosure also provide a computer-readable storage medium, which may store a computer program. When the computer program is executed by a processor, the processor implements the image generation model training method provided by the embodiments of the present disclosure.

[0131] The above-mentioned storage medium may, for example, include a memory 820 of computer program instructions, and the above-mentioned instructions may be executed by the processor 810 of the electronic device to complete the image generation model training method provided in the embodiment of the present disclosure. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (Random Access Memory, RAM), a compact disc read-only memory (Compact Disc ROM, CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0132] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an image generation model, characterized in that: include: Acquire a training sample set, wherein each training sample in the training sample set includes an image sketch, text, and a real image, and the text is used to describe information of the real image; For each training sample, the real image and the text are input into a pre-trained diffusion model, and the pre-trained diffusion model generates a predicted image based on the real image and the text; Fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model; Fine-tune the first diffusion model based on the image sketch to obtain a second diffusion model, and determine the second diffusion model as a trained diffusion model.

2. The method according to claim 1, characterized in that The step of inputting the real image and the text into a pre-trained diffusion model, and generating a predicted image based on the real image and the text by the pre-trained diffusion model, comprises: Inputting the text into the text encoder of the pre-trained diffusion model, and encoding the text by the text encoder to obtain a text feature vector; Inputting the real image into the autoencoder of the pre-trained diffusion model, and encoding the real image by the autoencoder to obtain a first image feature vector; Inputting the first image feature vector into a noise adding neural network, and performing noise adding processing on the first image feature vector by the noise adding neural network to obtain a second image feature vector; The second image feature vector and the text feature vector are concatenated to obtain a first fused feature vector, the first fused feature vector is input into a denoising neural network, the denoising neural network performs denoising on the first fused feature vector, and the decoder is used to decode the target fused feature vector obtained after the denoising to generate the predicted image.

3. The method according to claim 1, characterized in that The fine-tuning and training of the first diffusion model based on the image sketch to obtain a second diffusion model includes: Inputting the image sketch into a preset adapter, encoding the image sketch based on the preset adapter, and obtaining a target image feature vector corresponding to the image sketch; Performing vector concatenation processing on the target image feature vector and the text feature vector corresponding to the text to obtain a second fused feature vector; The second fused feature vector is input into the first diffusion model, and the first diffusion model is fine-tuned and trained to obtain the second diffusion model.

4. The method according to claim 3, characterized in that Before inputting the image sketch into a preset adapter and encoding the image sketch based on the preset adapter to obtain a target image feature vector corresponding to the image sketch, the method further includes: Performing pixel rearrangement processing on the image sketch to obtain a target sketch; The step of inputting the image sketch into a preset adapter, encoding the image sketch based on the preset adapter, and obtaining a target image feature vector corresponding to the image sketch includes: Input the target sketch into the first branch of the preset adapter, and the first branch encodes the target sketch to obtain a third image feature vector, and input the target sketch into the second branch of the preset adapter, and the second branch encodes the target sketch to obtain a fourth image feature vector; The third image feature vector and the fourth image feature vector are concatenated to obtain the target image feature vector.

5. The method according to claim 4, characterized in that The first branch consists of a first convolutional layer and three first encoders, the second branch consists of a second convolutional layer and three second encoders, the convolution kernel of the first convolutional layer is larger than the convolution kernel of the second convolutional layer; wherein the first encoder includes three residual modules and a pooling layer, and the second encoder includes three residual modules, a third convolutional layer and a fourth convolutional layer.

6. An image generation method, characterized in that: include: Obtaining a sketch to be processed and text to be processed, wherein the text to be processed is used to describe information of an image to be generated; The sketch to be processed and the text to be processed are input into a trained diffusion model, and the trained diffusion model generates a target image based on the sketch to be processed and the text to be processed. The trained diffusion model is trained using the image generation model training method described in any one of claims 1 to 5.

7. An image generation model training device, characterized in that: include: A training set acquisition unit, used to acquire a training sample set, each training sample in the training sample set includes an image sketch, text and a real image, and the text is used to describe information of the real image; A first training unit is used for inputting the real image and the text into a pre-trained diffusion model for each training sample, so that the pre-trained diffusion model generates a predicted image based on the real image and the text; Fine-tune the pre-trained diffusion model based on the predicted image and the real image to obtain a first diffusion model; The second training unit is used to fine-tune the first diffusion model based on the image sketch to obtain a second diffusion model, and determine the second diffusion model as the trained diffusion model.

8. An image generating device, characterized in that: include: A data acquisition unit, used to acquire a sketch to be processed and a text to be processed, wherein the text to be processed is used to describe information of an image to be generated; An image generation unit is used to input the sketch to be processed and the text to be processed into a trained diffusion model, and the trained diffusion model generates a target image based on the sketch to be processed and the text to be processed, and the trained diffusion model is trained using the image generation model training method described in any one of claims 1 to 5.

9. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; Wherein, the processor is used to read the executable instructions from the memory and execute the executable instructions to implement the image generation model training method described in any one of claims 1 to 5 or the image generation method described in claim 6.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the processor implements the image generation model training method described in any one of claims 1 to 5 or the image generation method described in claim 6.

Images