Method for constructing image generation model, image generation method and device thereof

Abstract

The present disclosure provides a method for constructing an image generation model, an image generation method and an apparatus thereof, and relates to the field of image processing, in particular to the field of artificial intelligence. The specific implementation scheme is: obtaining a sample image and a sample prompt corresponding to the sample image; processing the sample image based on a first preprocessing network to obtain a control condition and a target latent space code corresponding to the sample image; converting the sample prompt into a text vector based on a second preprocessing network; taking the target latent space code, the text vector and the control condition as inputs of a UNet network to be trained, training the UNet network in combination with a noise distribution output by the UNet network to obtain a trained target UNet network; and constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, in particular to the field of artificial intelligence, and more particularly to a method for constructing an image generation model, an image generation method and an apparatus thereof. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, image generation models have been widely applied in multiple fields. The core of image generation technology is to automatically generate image content related to the input user image through an algorithm model (for example, generating a specific style of portrait image based on the user uploaded image after the user uploads the image). In related technologies, when generating images, the image generation model is difficult to ensure that the generated content is highly consistent with the specific features of the user image input by the user, especially when it involves detailed features (such as the image of a person and the posture). SUMMARY

[0003] The present disclosure provides a method for constructing an image generation model, an image generation method, an apparatus, a device and a storage medium.

[0004] According to an aspect of the present disclosure, a method for constructing an image generation model is provided, which comprises: obtaining a sample image and a sample prompt word corresponding to the sample image; processing the sample image based on a first preprocessing network to obtain a control condition corresponding to the sample image and a target latent space encoding; converting the sample prompt word into a text vector based on a second preprocessing network; taking the target latent space encoding, the text vector and the control condition as inputs of a UNet network to be trained, and training the UNet network in combination with a noise distribution output by the UNet network to obtain a trained target UNet network; and constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder.

[0005] In the present application, the control condition is used as input information throughout the UNet network, so that a better target UNet network is obtained after training. The image generation model constructed based on the target UNet network can significantly improve the quality of the image generated by the image generation model, so that the finally generated image can highly represent the user features of the image input by the user.

[0006] According to another aspect of the present disclosure, an image generation method is provided, which comprises: obtaining an initial image input by a user and a prompt word; inputting the initial image into a first preprocessing network of an image generation model and inputting the prompt word into a second preprocessing network of the image generation model to obtain a target image output by the image generation model; wherein the image generation model is generated by using the method for constructing an image generation model as described in the above embodiments.

[0007] According to another aspect of the present disclosure, a device for constructing an image generation model is provided, comprising: an acquisition module configured to acquire a sample image and a sample prompt corresponding to the sample image; a first processing module configured to process the sample image based on a first pre-processing network to obtain a control condition and a target latent space code corresponding to the sample image; a second processing module configured to convert the sample prompt into a text vector based on a second pre-processing network; a training module configured to take the target latent space code, the text vector, and the control condition as inputs of a UNet network to be trained, and train the UNet network in combination with a noise distribution output by the UNet network to obtain a trained target UNet network; and a generation module configured to construct an image generation model based on the first pre-processing network, the second pre-processing network, the target UNet network, and a decoder.

[0008] According to another aspect of the present disclosure, an image generation device is provided, comprising: an acquisition module configured to acquire an initial image and a prompt input by a user; and a generation module configured to input the initial image into a first pre-processing network of an image generation model, input the prompt into a second pre-processing network of the image generation model, and acquire a target image output by the image generation model; wherein the image generation model is generated by the device for constructing an image generation model according to the above-mentioned embodiments.

[0009] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method for constructing an image generation model according to the first embodiment or the image generation method according to the second embodiment.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to perform the method for constructing an image generation model according to the first embodiment or the image generation method according to the second embodiment.

[0011] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method for constructing an image generation model according to the first embodiment or the image generation method according to the second embodiment.

[0012] It should be understood that the content described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings are used to better understand the present scheme and do not constitute a limitation on the present disclosure. Among them:

[0014] Figure 1 is a schematic diagram of an exemplary implementation of a method for constructing an image generation model according to an exemplary embodiment of the present disclosure.

[0015] Figure 2 is a schematic diagram of a method for constructing a UNet network to be trained according to an exemplary embodiment of the present disclosure.

[0016] Figure 3 is a schematic diagram of an initial UNet network according to an exemplary embodiment of the present disclosure.

[0017] Figure 4 is a schematic diagram of an adapter module according to an exemplary embodiment of the present disclosure.

[0018] Figure 5 is a schematic diagram of an image generation model according to an exemplary embodiment of the present disclosure.

[0019] Figure 6 is a schematic diagram of an exemplary implementation of a method for constructing an image generation model according to an exemplary embodiment of the present disclosure.

[0020] Figure 7 is a schematic diagram of an exemplary implementation of an image generation method according to an exemplary embodiment of the present disclosure.

[0021] Figure 8 is a schematic diagram of a construction device of an image generation model according to an exemplary embodiment of the present disclosure.

[0022] Figure 9 is a schematic diagram of an image generation device according to an exemplary embodiment of the present disclosure.

[0023] Figure 10 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, descriptions of well-known functions and structures are omitted in the following description.

[0025] Image Processing, the technology of using computer to analyze image to achieve the desired results. Also known as image processing. Image processing generally refers to digital image processing. Digital image refers to a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other devices. The elements of the array are called pixels, and their values are called gray values. Image processing technology generally includes image compression, enhancement and restoration, matching, description and identification of three parts.

[0026] Artificial Intelligence (AI), is a discipline that studies making computers simulate some thinking processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.) of human life, both hardware and software technologies. Artificial intelligence hardware technology generally includes computer vision technology, speech recognition technology, natural language processing technology, and learning / deep learning, big data processing technology, knowledge graph technology, etc.

[0027] In the technical solution of the present disclosure, the acquisition, storage and application of user personal information involved comply with relevant laws and regulations and do not violate public order and good customs.

[0028] Figure 1 is an exemplary embodiment of a method for constructing an image generation model shown in the present application, as shown in Figure 1 The method for constructing the image generation model comprises the following steps:

[0029] S101, acquiring sample images and sample prompt words corresponding to the sample images.

[0030] First, collect sample images, such as sample images of people.

[0031] The sample images are training images used to train the U-shaped network (UNet) network to be trained, and each sample image has its corresponding sample prompt word. The sample prompt word is used to indicate how the image generation model converts the image.

[0032] For example, a sample image is a modern woman image, and its corresponding sample prompt word is "ancient costume", "building background", etc. The meaning is: convert the sample image into an ancient costume image with a building background, while the pre-set character features (such as face or posture, etc.) in the sample image remain unchanged. For example, when generating a character photo, the facial features do not change, but other features such as character background and hair accessories will change.

[0033] S102, processing the sample images based on a first pre-processing network to obtain control conditions and target latent space encodings corresponding to the sample images.

[0034] The first pre-processing network is used to process the sample image to obtain a depth map corresponding to the sample image, and latent space encoding corresponding to the depth map is used as a control condition of the UNet network to be trained.

[0035] The first pre-processing network is used to process the sample image (for example, noise processing after encoding) to obtain target latent space encoding corresponding to the sample image.

[0036] S103, the sample prompt is converted into a text vector based on the second pre-processing network.

[0037] The second pre-processing network includes a text encoder for converting the sample prompt input into a text vector.

[0038] S104, the target latent space encoding, the text vector and the control condition are used as inputs of the UNet network to be trained, and the UNet network is trained in combination with the noise distribution output by the UNet network to obtain a trained target UNet network.

[0039] The target latent space encoding, the text vector and the control condition are used as inputs of the UNet network to be trained, the loss value of the UNet network is calculated in combination with the noise distribution output by the UNet network, and the UNet network is iteratively trained based on the loss value to obtain a target UNet network generated after training.

[0040] S105, an image generation model is constructed based on the first pre-processing network, the second pre-processing network, the target UNet network and the decoder.

[0041] The first pre-processing network and the second pre-processing network are both constructed before the target UNet network, and the outputs of the first pre-processing network and the second pre-processing network are both directed to the target UNet network.

[0042] The target UNet network is mainly used to predict a noise distribution, and based on the target latent space encoding input into the target UNet network, the noise distribution is subtracted, and then the noise distribution of the next time step is predicted. After the target UNet network is denoised for multiple time steps, the output of the target UNet network is obtained.

[0043] The encoder is constructed after the target UNet network. The encoder is used to convert the output of the target UNet network after denoising for multiple time steps into a final generated image.

[0044] The embodiment of the application proposes a method for constructing an image generation model, which comprises the following steps: obtaining a sample image and a sample prompt corresponding to the sample image; processing the sample image based on a first preprocessing network to obtain a control condition and a target latent space code corresponding to the sample image; converting the sample prompt into a text vector based on a second preprocessing network; taking the target latent space code, the text vector and the control condition as inputs of a UNet network to be trained, training the UNet network in combination with a noise distribution output by the UNet network, and obtaining a trained target UNet network; and constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder. In the application, the control condition is used as input information throughout the UNet network, so that a better target UNet network is obtained after training. The image generation model constructed based on the target UNet network can significantly improve the quality of the image generated by the image generation model, so that the finally generated image can highly represent the user features of the image input by the user.

[0045] In the application, the UNet network to be trained involved in the above embodiment is generated based on improvement of the UNet network used in the conventional technology. The specific improvement scheme can be summarized as follows: an adapter module designed in advance is coupled on an initial UNet network to obtain the UNet network to be trained, and then the UNet network to be trained is trained based on the text vector and the control condition. Figure 2 Further, the method for constructing the UNet network to be trained comprises the following steps:

[0046] S201, determining a target Transformer module from candidate Transformer modules contained in a down-sampling layer, an up-sampling layer and an intermediate layer of an initial UNet network.

[0047] The initial UNet network in the application is often applied in a Stable Diffusion model, Figure 3 is a schematic diagram of an initial UNet network shown in the application.

[0048] As Figure 3 shown, the initial UNet network comprises a down-sampling layer, an up-sampling layer and an intermediate layer, Figure 3 res on the left represents a residual module, Figure 3 d1, d2, d3 and d4 on the left represent the first, second, third and fourth down-sampling layers respectively, Figure 3The u1, u2, u3 and u4 on the right represent the up-sampling of the first, second, third and fourth layers in sequence. The first, second and third layers on the left include two residual modules and two Transformer modules, the second, third and fourth layers on the right include three residual modules and three Transformer modules, and the middle layer includes two residual modules and one Transformer module connected together. The fourth layer on the left includes two residual modules and does not introduce a Transformer module; the first layer on the right includes three residual modules and does not introduce a Transformer module. Each layer of the down-sampling on the left is additionally connected to the corresponding layer on the right, so as to splice the two networks.

[0049] In some implementable embodiments, a target Transformer module can be selected from all the above-mentioned Transformer modules according to actual conditions, and the target Transformer module can be one or more.

[0050] In some implementable embodiments, the first Transformer module included in each of the first, second and third layers of the down-sampling on the left is taken as a target Transformer module, the first Transformer module included in each of the second, third and fourth layers of the up-sampling on the right is taken as a target Transformer module, and the Transformer module included in the middle layer is taken as a target Transformer module, that is, a total of seven target Transformer modules are determined (in the figure, the seven target Transformer modules are shaded and blackened). Figure 3

[0051] S202, coupling an adapter module between a self-attention module included in a target Transformer module and a previous layer of the self-attention module, to obtain a UNet network to be trained.

[0052] Each of the above-mentioned target Transformer modules includes a self-attention module, in the present application, an adapter module is designed, and the adapter module is coupled between the self-attention module included in each target Transformer module and the previous layer of the self-attention module, to obtain a UNet network to be trained.

[0053] ​It is not difficult to understand that the above determines 7 target Transformer modules as an example, and the adapter module is coupled between the self-attention module included in each of the 7 target Transformer modules and the upper layer of the self-attention module. That is, 7 adapter modules are coupled in the initial UNet network to obtain the UNet network to be trained.

[0054] The input of the adapter module includes the control condition determined above, and the adapter module is used to integrate the control condition into the UNet network.

[0055] In the UNet network to be trained, the input of the adapter module is the output of the upper layer and the control condition; the output of the adapter module and the output of the upper layer are spliced as the input of the self-attention module.

[0056] In this application, the upper layer refers to the upper layer of the self-attention module before being coupled into the adapter module, which is uniformly referred to as the upper layer in this application. In the actual UNet network structure, the upper layer of the self-attention module is the normalization layer.

[0057] In this way, during training, the control condition can be input information throughout the UNet network to be trained, so that a better target UNet network is obtained after training.

[0058] For easy understanding, Figure 4 is a schematic diagram of an adapter module shown in this application, as Figure 4 The adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, wherein:

[0059] The resolution alignment unit includes a plurality of convolution kernels, which are used to align the spatial resolution of the control condition and the output of the upper layer.

[0060] The second splicing unit is used to splice the control condition after spatial resolution alignment and the output of the upper layer to obtain spliced features.

[0061] The feature aggregation unit includes a plurality of residual blocks, which are used to fuse the spliced features to obtain fused features.

[0062] The spatial attention unit includes a plurality of convolution layers and one Sigmoid activation function, and the output of the Sigmoid activation function is a result between 0 and 1. The spatial attention unit is used to generate a weighted parameter corresponding to the fused features based on the spliced features.

[0063] The multiplication unit is configured to multiply the fused feature and the weighting parameter, and a result of the multiplication is taken as an output of the adapter module.

[0064] The above describes in detail the construction method of the UNet network to be trained, details the input and output of the adapter module designed by the application and the role of each unit, and determines the specific position of the adapter module designed by the application in the UNet network to be trained, so that the control condition can be used as input information throughout the UNet network to be trained, and a better target UNet network is obtained after training.

[0065] Further, in some implementations, when the UNet network to be trained is constructed, a pre-designed low-rank adapter (Low-Rank Adaptation, LORA) can also be coupled in the self-attention module and the interaction attention module contained in the target Transformer module. That is, taking the above-mentioned determination of 7 target Transformer modules as an example, each target Transformer module contains a self-attention module and an interaction attention module, and in this application, the LORA adapter is coupled in the self-attention module and the LORA adapter is coupled in the interaction attention module.

[0066] The LORA adapter usually only increases a small number of parameters, which are specially used to adjust the attention mechanism (self-attention and interaction attention) of the UNet network. Compared with directly training all parameters of the entire UNet network to be trained, this method greatly reduces the number of parameters to be trained; during the training process, the LoRA adapter can adjust the feature representation of the UNet network, thereby improving the adaptability of the UNet network to different control conditions.

[0067] For ease of understanding, Figure 5 is a schematic diagram of an image generation model shown in the application, as Figure 5 As shown in the figure, the image generation model is constructed and generated by a first preprocessing network, a second preprocessing network, a target UNet network and a decoder.

[0068] The first preprocessing network includes a depth map estimation unit, a first encoder, a second encoder, a noise adding unit and a first splicing unit.

[0069] After the sample image is input into the first preprocessing network, the first preprocessing network performs depth map estimation on the sample image based on a depth map estimation unit to obtain a depth map of the sample image; and then encodes the depth map based on a first encoder to obtain a first latent space code, and inputs the first latent space code as a control condition. It is not difficult to understand that the depth map provides the geometric structure information of the objects in the image, and reflects the relative distance and position between different objects in the scene. Inputting the first latent space code corresponding to the depth map as the control condition can ensure that the image generated by the image generation model is consistent with the original image in terms of spatial structure and perspective.

[0070] Further, the first preprocessing network encodes the sample image based on a second encoder to obtain a second latent space code; adds noise to the second latent space code based on a noise adding unit to obtain a third latent space code; and splices the third latent space code and the first latent space code based on a first splicing unit to obtain a target latent space code.

[0071] The first latent space code captures the geometric and spatial structure information of the image through the encoding of the depth map. The depth map itself contains the three-dimensional structure of the scene, and provides strong support for the position, shape, size, etc. of the objects in the image; the second latent space code comes from the image itself, and contains the visual features such as texture, color, background, etc. of the image. The two represent the spatial information and visual information of the image respectively. By combining these two kinds of information and splicing them, the target latent space code obtained can contain both the geometric (depth) information and the visual (texture, contour, etc.) information of the image, and this combination improves the comprehensive understanding and generation ability of the model for the image. Adding noise to the second latent space code through the noise adding unit can increase the robustness of the model.

[0072] The second preprocessing network includes a text encoder configured to convert the sample prompt into a text vector.

[0073] After obtaining the target latent space code, the text vector and the control condition, the target latent space code, the text vector and the control condition are input into the UNet network to be trained, and the UNet network is trained in combination with the noise distribution output by the UNet network to obtain a trained target UNet network. The control condition is directly input into each adapter module of the UNet network to be trained. For the relationship between the UNet network to be trained and the adapter module, see the detailed description above.

[0074] The decoder is configured to convert the output of the target UNet network after denoising for a plurality of time steps into a final generated image.

[0075] Figure 6is a schematic diagram of an exemplary embodiment of a method for constructing an image generation model shown in the present application, as Figure 6 The method for constructing an image generation model includes the following steps:

[0076] S601, determining a target Transformer module from candidate Transformer modules contained in a down-sampling layer, an up-sampling layer and an intermediate layer of an initial UNet network.

[0077] S602, coupling an adapter module between a self-attention module contained in the target Transformer module and an upper layer of the self-attention module.

[0078] S603, coupling a pre-designed LORA adapter between the self-attention module contained in the target Transformer module and an interaction attention module, to obtain a UNet network to be trained.

[0079] For specific implementation of steps S601-S603, refer to the specific description of the relevant part in the above embodiment, which will not be repeated here.

[0080] S604, obtaining a sample image and a sample prompt corresponding to the sample image.

[0081] S605, processing the sample image based on a first pre-processing network to obtain a control condition and a target latent space encoding corresponding to the sample image.

[0082] S606, converting the sample prompt into a text vector based on a second pre-processing network.

[0083] S607, taking the target latent space encoding, the text vector and the control condition as inputs of the UNet network to be trained, and training the UNet network in combination with a noise distribution output by the UNet network to obtain a trained target UNet network.

[0084] S608, constructing an image generation model based on the first pre-processing network, the second pre-processing network, the target UNet network and a decoder.

[0085] For specific implementation of steps S604-S608, refer to the specific description of the relevant part in the above embodiment, which will not be repeated here.

[0086] The original initial UNet network is improved in the embodiment of the application, the adapter module and the LORA adapter are coupled, and the control condition obtained based on the sample image is taken as input information throughout the UNet network, so that a more high-quality target UNet network is obtained after training, and the image generation model constructed based on the target UNet network can significantly improve the quality of the image finally generated by the image generation model, so that the finally generated image can highly represent the user features of the image input by the user.

[0087] Figure 7 is a schematic diagram of an exemplary embodiment of an image generation method shown in the application, as Figure 7 shown, the image generation method comprises the following steps:

[0088] S701, obtaining an initial image and a prompt word input by a user.

[0089] When used on a user line, the initial image and the prompt word input by the user are obtained.

[0090] S702, inputting the initial image to a first preprocessing network of an image generation model and inputting the prompt word to a second preprocessing network of the image generation model, and obtaining a target image output by the image generation model.

[0091] The image generation model is generated by using the construction method of the image generation model as introduced in the above embodiment.

[0092] In the embodiment of the application, when used on a user line, the initial image and the prompt word input by the user are input to the image generation model generated above, and the target image output by the image generation model can highly keep consistent with the initial image input by the user in geometric information and visual information, and at the same time, the target image output by the image generation model can be stylized based on the prompt word, which is more in line with the needs of the user and improves the user satisfaction.

[0093] Figure 8 is a schematic diagram of a construction device of an image generation model shown in the application, as Figure 8 shown, the construction device 800 of the image generation model comprises an acquisition module 801, a first processing module 802, a second processing module 803, a training module 804 and a generation module 805, wherein:

[0094] The acquisition module 801 is configured to acquire a sample image and a sample prompt word corresponding to the sample image.

[0095] The first processing module 802 is configured to process the sample image based on a first preprocessing network to obtain a control condition and a target latent space code corresponding to the sample image.

[0096] The second processing module 803 is configured to convert the sample prompt word into a text vector based on the second pre-processing network.

[0097] The training module 804 is configured to take the target latent space code, the text vector, and the control condition as input of the UNet network to be trained, and train the UNet network based on the noise distribution output by the UNet network to obtain the trained target UNet network.

[0098] The generation module 805 is configured to construct an image generation model based on the first pre-processing network, the second pre-processing network, the target UNet network, and the decoder.

[0099] In the device, the control condition is used as input information throughout the target UNet network, so that a better target UNet network is obtained after training, and the image generation model constructed based on the target UNet network can significantly improve the quality of the image generated by the image generation model, so that the finally generated image can highly represent the user features of the image input by the user.

[0100] Further, the first pre-processing network includes a depth map estimation unit and a first encoder, and the first processing module 802 is further configured to perform depth map estimation on the sample image based on the depth map estimation unit to obtain a depth map of the sample image, and perform encoding on the depth map based on the first encoder to obtain the first latent space code and take the first latent space code as the control condition.

[0101] Further, the first pre-processing network further includes a second encoder, a noise adding unit, and a first splicing unit, and the first processing module 802 is further configured to perform encoding on the sample image based on the second encoder to obtain a second latent space code, perform noise adding on the second latent space code based on the noise adding unit to obtain a third latent space code, and perform splicing on the third latent space code and the first latent space code based on the first splicing unit to obtain the target latent space code.

[0102] Further, the image generation model construction device 800 further includes a UNet network construction module, and the UNet network construction module is configured to couple a pre-designed adapter module to an initial UNet network to obtain the UNet network to be trained, wherein the input of the adapter module includes the control condition, and the adapter module is used to integrate the control condition into the UNet network.

[0103] Furthermore, the UNet network construction module is also used to: determine the target Transformer module from the candidate Transformer modules contained in the downsampling layer, upsampling layer and intermediate layer of the initial UNet network; couple the adapter module between the self-attention module contained in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

[0104] Furthermore, in the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated as the input of the self-attention module.

[0105] Furthermore, the adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, wherein: the resolution alignment unit includes multiple convolution kernels for aligning the spatial resolution of the control condition with the output of the previous layer; the second splicing unit is used to splice the control condition after spatial resolution alignment with the output of the previous layer to obtain splicing features; the feature aggregation unit includes multiple residual blocks for fusing the splicing features to obtain fused features; the spatial attention unit is used to generate weighting parameters corresponding to the fused features based on the splicing features; the multiplication unit is used to multiply the fused features and the weighting parameters, wherein the result of the multiplication is used as the output of the adapter module.

[0106] Furthermore, the UNet network building module is also used to couple the pre-designed LoRa adapter to the self-attention module and the interactive attention module contained in the target Transformer module.

[0107] Furthermore, the decoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

[0108] Figure 9 This is a schematic diagram of an image generating device shown in the present application. Figure 9 As shown, the image generating device 900 includes:

[0109] The acquisition module 901 is used to acquire the initial image and prompt word input by the user.

[0110] The generation module 902 is used to input the initial image into the first preprocessing network of the image generation model, input the prompt word into the second preprocessing network of the image generation model, and obtain the target image output by the image generation model.

[0111] The image generation model is generated using the image generation model construction device introduced in the above embodiment.

[0112] The device, when used on a user line, inputs the initial image and the prompt word input by the user into the generated image generation model, and the target image output by the image generation model can be highly consistent with the initial image input by the user in geometric information and visual information, and can be stylized based on the prompt word, so that the target image output by the image generation model is more in line with the needs of the user, and the user satisfaction is improved.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0114] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0115] As shown in Figure 10 The device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded into a random access memory (RAM) 1003 from a storage unit 1008. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0116] Various components in the device 1000 are connected to the I / O interface 1005, including an input unit 1006, such as a keyboard, a mouse, etc., an output unit 1007, such as various types of displays, speakers, etc., a storage unit 1008, such as a magnetic disk, an optical disk, etc., and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0117] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs various methods and processes described above, such as the construction method of an image generation model, the image generation method. For example, in some embodiments, the construction method of an image generation model, the image generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded onto the RAM 1003 and executed by the computing unit 1001, one or more steps of the construction method of an image generation model, the image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the construction method of an image generation model, the image generation method by any other suitable means, such as by means of firmware.

[0118] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0119] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or the block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine and partially on a remote machine or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0124] It should be understood that the various forms of flow shown above can be re-ordered, added to, or have steps deleted, using the flow. For example, the steps described in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure can be achieved, which are not limited herein.

[0125] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for constructing an image generation model, comprising: Obtaining a sample image and a sample prompt word corresponding to the sample image; performing depth map estimation on the sample image based on a first preprocessing network to obtain a depth map of the sample image, and encoding the depth map to obtain a first latent space encoding as a control condition; Encoding the sample image based on the first preprocessing network to obtain a second latent space code and then adding noise to obtain a third latent space code, and concatenating the third latent space code and the first latent space code to obtain a target latent space code; Converting the sample prompt word into a text vector based on a second preprocessing network; Coupling a pre-designed adapter module to the initial UNet network to obtain a UNet network to be trained, wherein the input of the adapter module includes the control condition, and the adapter module is used to integrate the control condition into the UNet network; Taking the target latent space encoding, the text vector, and the control condition as inputs of the UNet network, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; An image generation model is constructed based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

2. The method according to claim 1, wherein The pre-designed adapter module is coupled to the initial UNet network to obtain the UNet network to be trained, including: Determine a target Transformer module from the candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network; The adapter module is coupled between the self-attention module included in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

3. The method according to claim 2, wherein: In the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated and used as the input of the self-attention module.

4. The method according to claim 3, wherein: The adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit, and a multiplication unit, wherein: The resolution alignment unit includes a plurality of convolution kernels for aligning the spatial resolution of the control condition with the output of the previous layer; The second splicing unit is used to splice the control condition after spatial resolution alignment with the output of the previous layer to obtain a splicing feature; The feature aggregation unit includes a plurality of residual blocks, which are used to fuse the spliced ​​features to obtain fused features; The spatial attention unit is used to generate a weighting parameter corresponding to the fusion feature based on the splicing feature; The multiplication unit is used to multiply the fusion feature and the weighting parameter, wherein the multiplication result is used as the output of the adapter module.

5. The method according to claim 4, wherein The method further comprises: A pre-designed LoRa adapter is coupled to the self-attention module and the interactive attention module contained in the target Transformer module.

6. The method according to claim 1, wherein The decoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

7. A method for generating an image, comprising: Get the initial image and prompt word entered by the user; Inputting the initial image into a first preprocessing network of an image generation model, inputting the prompt word into a second preprocessing network of the image generation model, and obtaining a target image output by the image generation model; The image generation model is generated by the method according to any one of claims 1 to 6.

8. A device for constructing an image generation model, comprising: An acquisition module, configured to acquire a sample image and a sample prompt word corresponding to the sample image; A first processing module is configured to perform depth map estimation on the sample image based on a first preprocessing network to obtain a depth map of the sample image, and encode the depth map to obtain a first latent space code as a control condition; Encoding the sample image based on the first preprocessing network to obtain a second latent space code, then adding noise to obtain a third latent space code, and concatenating the third latent space code and the first latent space code to obtain a target latent space code; A second processing module, configured to convert the sample prompt word into a text vector based on a second preprocessing network; A UNet network construction module is used to couple a pre-designed adapter module to the initial UNet network to obtain a UNet network to be trained, wherein the input of the adapter module includes the control condition, and the adapter module is used to integrate the control condition into the UNet network; A training module, configured to use the target latent space encoding, the text vector, and the control condition as inputs of the UNet network, and train the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; A generation module is used to build an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

9. The device according to claim 8, wherein The UNet network building module is also used to: Determine a target Transformer module from the candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network; The adapter module is coupled between the self-attention module included in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

10. The device according to claim 9, wherein In the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated and used as the input of the self-attention module.

11. The device according to claim 10, wherein The adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit, and a multiplication unit, wherein: The resolution alignment unit includes a plurality of convolution kernels for aligning the spatial resolution of the control condition with the output of the previous layer; The second splicing unit is used to splice the control condition after spatial resolution alignment with the output of the previous layer to obtain a splicing feature; The feature aggregation unit includes a plurality of residual blocks, which are used to fuse the spliced ​​features to obtain fused features; The spatial attention unit is used to generate a weighting parameter corresponding to the fusion feature based on the splicing feature; The multiplication unit is used to multiply the fusion feature and the weighting parameter, wherein the multiplication result is used as the output of the adapter module.

12. The device according to claim 11, wherein The UNet network building module is also used to: A pre-designed LoRa adapter is coupled to the self-attention module and the interactive attention module contained in the target Transformer module.

13. The device according to claim 8, wherein The decoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

14. An image generating device, wherein: include: An acquisition module is used to obtain the initial image and prompt words input by the user; a generation module, configured to input the initial image into a first preprocessing network of an image generation model, input the prompt word into a second preprocessing network of the image generation model, and obtain a target image output by the image generation model; The image generation model is generated by the device described in any one of claims 8 to 13.

15. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6 or claim 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6 or claim 7.

17. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 6 or claim 7.

Images