Construction method of image generation model, image generation method and device thereof

Abstract

The invention provides an image generation model construction method, an image generation method and an image generation device, and relates to the field of image processing, in particular to the field of artificial intelligence. According to the specific implementation scheme, a sample image and a sample cue word corresponding to the sample image are obtained; processing the sample image based on a first preprocessing network to obtain a control condition corresponding to the sample image and a target potential space code; converting the sample cue word into a text vector based on a second preprocessing network; the target potential space codes, the text vectors and the control conditions serve as input of a UNet network to be trained, the UNet network is trained in combination with noise distribution output by the UNet network, and a trained target UNet network is obtained; and constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, in particular to the field of artificial intelligence, and specifically to a method for constructing an image generation model, an image generation method and a device thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, image generation models have been widely used in many fields. The core of image generation technology is to automatically generate image content related to the input user image through an algorithm model (for example, after the user uploads the image, a photo image of a specific style is generated based on the user's uploaded image). In related technologies, when generating images, it is difficult for image generation models to ensure that the generated content is highly consistent with the specific features of the user image input by the user, especially when it involves detailed features (such as character image and posture). Summary of the invention

[0003] The present disclosure provides a method for constructing an image generation model, an image generation method, an apparatus, a device, and a storage medium.

[0004] According to one aspect of the present disclosure, a method for constructing an image generation model is provided, which comprises obtaining a sample image and a sample prompt word corresponding to the sample image; processing the sample image based on a first preprocessing network to obtain a control condition and a target latent space encoding corresponding to the sample image; converting the sample prompt word into a text vector based on a second preprocessing network; using the target latent space encoding, the text vector and the control condition as inputs of a UNet network to be trained, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; and constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder.

[0005] In the present application, by running the control conditions as input information through the UNet network, a better target UNet network is obtained after training. The image generation model constructed based on the target UNet network can significantly improve the quality of the image finally generated by the image generation model, so that the finally generated image can highly represent the user characteristics of the image input by the user.

[0006] According to another aspect of the present disclosure, an image generation method is provided, comprising: obtaining an initial image and a prompt word input by a user; inputting the initial image into a first preprocessing network of an image generation model, inputting the prompt word into a second preprocessing network of the image generation model, and obtaining a target image output by the image generation model; wherein the image generation model is generated using the image generation model construction method described in the above embodiment.

[0007] According to another aspect of the present disclosure, a device for constructing an image generation model is provided, including: an acquisition module, used to acquire a sample image and a sample prompt word corresponding to the sample image; a first processing module, used to process the sample image based on a first preprocessing network, and obtain a control condition and a target latent space encoding corresponding to the sample image; a second processing module, used to convert the sample prompt word into a text vector based on a second preprocessing network; a training module, used to use the target latent space encoding, the text vector and the control condition as inputs of a UNet network to be trained, and train the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; a generation module, used to construct an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder.

[0008] According to another aspect of the present disclosure, an image generating device is provided, comprising: an acquisition module, used to acquire an initial image and a prompt word input by a user; a generation module, used to input the initial image into a first preprocessing network of an image generating model, input the prompt word into a second preprocessing network of the image generating model, and acquire a target image output by the image generating model; wherein the image generating model is generated using the image generating model constructing device introduced in the above-mentioned embodiment.

[0009] According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for constructing an image generation model introduced in the first embodiment or the image generation method introduced in the second embodiment.

[0010] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the image generation model construction method introduced in the first embodiment or the image generation method introduced in the second embodiment.

[0011] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for constructing an image generation model introduced in the first embodiment or the image generation method introduced in the second embodiment.

[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0014] Figure 1 It is a schematic diagram of an exemplary implementation of a method for constructing an image generation model according to an exemplary embodiment of the present disclosure.

[0015] Figure 2 It is a schematic diagram of a method for constructing a UNet network to be trained according to an exemplary embodiment of the present disclosure.

[0016] Figure 3 is a schematic diagram of an initial UNet network according to an exemplary embodiment of the present disclosure.

[0017] Figure 4 is a schematic diagram of an adapter module according to an exemplary embodiment of the present disclosure.

[0018] Figure 5 is a schematic diagram of an image generation model according to an exemplary embodiment of the present disclosure.

[0019] Figure 6 It is a schematic diagram of an exemplary implementation of a method for constructing an image generation model according to an exemplary embodiment of the present disclosure.

[0020] Figure 7 is a schematic diagram of an exemplary implementation of an image generating method according to an exemplary embodiment of the present disclosure.

[0021] Figure 8 It is a schematic diagram of a device for constructing an image generation model according to an exemplary embodiment of the present disclosure.

[0022] Fig. 9 is a schematic diagram of an image generating device according to an exemplary embodiment of the present disclosure.

[0023] Fig.10 is a schematic diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0025] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.

[0026] Artificial Intelligence (AI) is a discipline that studies how computers can simulate certain thought processes and intelligent behaviors of human beings (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level technologies and software-level technologies. AI hardware technologies generally include computer vision technology, speech recognition technology, natural language processing technology, as well as learning / deep learning, big data processing technology, knowledge graph technology, and other aspects.

[0027] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0028] Figure 1 is a schematic diagram of an exemplary implementation of a method for constructing an image generation model shown in the present application, such as Figure 1 As shown, the method for constructing the image generation model includes the following steps:

[0029] S101, obtaining a sample image and a sample prompt word corresponding to the sample image.

[0030] Here, sample images are first collected, for example, the sample images are images of people.

[0031] The sample images are training images used to train the U-shaped network (UNet) to be trained, and each sample image has its corresponding sample prompt word. The sample prompt word is used to indicate how the image generation model converts the image.

[0032] For example, a sample image is an image of a modern woman, and its corresponding sample prompt words are "ancient costume", "building background", etc. The meaning is: the sample image is converted into an ancient costume image with a building as the background, while the preset character features (such as face or posture) in the sample image remain unchanged. For example, when generating a portrait, the facial features do not change, but other features such as the character background and hair accessories will change.

[0033] S102, processing the sample image based on the first preprocessing network to obtain control conditions and target latent space encoding corresponding to the sample image.

[0034] Among them, the sample image can be processed based on the first preprocessing network to obtain a depth map corresponding to the sample image, and the latent space encoding corresponding to the depth map is used as the control condition of the UNet network to be trained. The control condition is a feature that is crucial to generating an image.

[0035] The sample image is processed based on the first preprocessing network (for example, noise is added after encoding) to obtain a target latent space encoding corresponding to the sample image.

[0036] S103, converting the sample prompt word into a text vector based on the second preprocessing network.

[0037] The second preprocessing network includes a text encoder for converting sample prompt words input into the second preprocessing network into text vectors.

[0038] S104, taking the target latent space encoding, the text vector and the control condition as the input of the UNet network to be trained, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network.

[0039] The target latent space encoding, text vector and control condition are used as the input of the UNet network to be trained. The loss value of the UNet network is calculated based on the noise distribution of the UNet network output. The UNet network is iteratively trained based on the loss value to obtain the target UNet network generated after the training is completed.

[0040] S105, constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

[0041] Among them, the first preprocessing network and the second preprocessing network are both constructed before the target UNet network, and the outputs of the first preprocessing network and the second preprocessing network both point to the target UNet network.

[0042] Among them, the target UNet network is mainly used to predict the noise distribution, and based on the target latent space encoding input to the target UNet network minus the noise distribution, the noise distribution of the next time step is predicted, and the output of the target UNet network is obtained after the target UNet network is denoised for multiple time steps.

[0043] The encoder is built after the target UNet network. The encoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

[0044] The embodiment of the present application proposes a method for constructing an image generation model, by obtaining a sample image and a sample prompt word corresponding to the sample image; processing the sample image based on a first preprocessing network to obtain a control condition and a target latent space encoding corresponding to the sample image; converting the sample prompt word into a text vector based on a second preprocessing network; using the target latent space encoding, the text vector and the control condition as the input of the UNet network to be trained, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder. In the present application, by running the control condition through the UNet network as input information, a higher quality target UNet network is obtained after training, and the image generation model constructed based on the target UNet network can significantly improve the quality of the image finally generated by the image generation model, so that the finally generated image can highly represent the user characteristics of the image input by the user.

[0045] In this application, the UNet network to be trained involved in the above embodiments is generated based on the improvement of the UNet network used in the traditional technology. The specific improvement scheme can be summarized as follows: the pre-designed adapter module is coupled to the initial UNet network to obtain the UNet network to be trained. Figure 2 The following further introduces the construction method of the UNet network to be trained. The construction method of the UNet network to be trained includes:

[0046] S201, determining a target Transformer module from candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network.

[0047] The initial UNet network in this application is often used in the Stable Diffusion model. Figure 3 It is a schematic diagram of an initial UNet network shown in this application.

[0048] like Figure 3 As shown, the initial UNet network includes downsampling layers, upsampling layers and intermediate layers. Figure 3 The res above represents the residual module, Figure 3 The d1, d2, d3, and d4 on the left represent the 1st, 2nd, 3rd, and 4th layers of downsampling, respectively. Figure 3The u1, u2, u3, and u4 on the right represent the 1st, 2nd, 3rd, and 4th layers of the upsampling, respectively. The 1st, 2nd, and 3rd layers of the left downsampling each include 2 residual modules and 2 Transformer modules, and the 2nd, 3rd, and 4th layers of the right upsampling each include 3 residual modules and 3 Transformer modules. The middle layer consists of 2 residual modules and 1 Transformer module. The 4th layer of the left downsampling contains two residual modules, and no Transformer module is introduced; the 1st layer of the right upsampling contains 3 residual modules, and no Transformer module is introduced. Each layer of the left downsampling is additionally connected to the corresponding layer of the right upsampling, and the two networks are spliced.

[0049] In some feasible embodiments, a target Transformer module may be selected from all the above-mentioned Transformer modules according to actual conditions, and the number of the target Transformer modules may be one or more.

[0050] In some achievable embodiments, the first Transformer module included in the left downsampling layer 1, layer 2, and layer 3 is used as the target Transformer module, and the first Transformer module included in the right upsampling layer 2, layer 3, and layer 4 is used as the target Transformer module, and the Transformer module included in the middle layer is used as the target Transformer module, that is, a total of 7 target Transformer modules are determined (in Figure 3 In the figure, these 7 target Transformer modules have been shaded).

[0051] S202, coupling the adapter module between the self-attention module included in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

[0052] Among them, each of the above-mentioned target Transformer modules includes a self-attention module. In this application, an adapter module is designed, and the adapter module is coupled between the self-attention module contained in each target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

[0053] It is not difficult to understand that, taking the above-mentioned 7 target Transformer modules as an example, this step couples the adapter modules between the self-attention modules contained in each of the 7 target Transformer modules and the upper layer of the self-attention modules. That is, 7 adapter modules are coupled in the initial UNet network to obtain the UNet network to be trained.

[0054] Among them, the input of the adapter module includes the control conditions determined above, and the adapter module is used to integrate the control conditions into the UNet network.

[0055] Among them, in the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated as the input of the self-attention module.

[0056] In this application, the previous layer refers to the previous layer of the self-attention module before it is coupled into the adapter module. It is uniformly referred to as the previous layer in this application. In the actual UNet network structure, the previous layer of the self-attention module is the normalization layer.

[0057] In this way, during training, the control conditions can be used as input information throughout the UNet network to be trained, so that a better target UNet network can be obtained after training.

[0058] For ease of understanding, Figure 4 is a schematic diagram of an adapter module shown in this application, such as Figure 4 As shown, the adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, where:

[0059] The resolution alignment unit includes multiple convolution kernels to align the spatial resolution of the control condition with the output of the previous layer.

[0060] The second splicing unit is used to splice the control condition after spatial resolution alignment with the output of the previous layer to obtain the splicing feature.

[0061] The feature aggregation unit includes multiple residual blocks, which are used to fuse the spliced ​​features to obtain fused features.

[0062] The spatial attention unit includes multiple convolutional layers and a Sigmoid activation function. The output of the Sigmoid activation function is a result between 0 and 1. The spatial attention unit is used to generate weighted parameters corresponding to fused features based on the splicing features.

[0063] The multiplication unit is used to multiply the fused features and the weighted parameters, where the multiplication result is used as the output of the adapter module.

[0064] The above details the construction method of the UNet network to be trained, the input and output of the adapter module designed in this application and the function of each unit, and clarifies the specific position of the adapter module designed in this application in the UNet network to be trained, so that the control conditions can be used as input information throughout the UNet network to be trained, so that a better target UNet network can be obtained after training.

[0065] Furthermore, in some implementations, when constructing the UNet network to be trained, a pre-designed low-rank adapter (LORA) can also be coupled to the self-attention module and the interactive attention module contained in the target Transformer module. That is, continuing to take the above-mentioned determination of 7 target Transformer modules as an example, each target Transformer module includes a self-attention module and an interactive attention module. In this application, a LORA adapter is coupled to both the self-attention module and the interactive attention module.

[0066] The LoRa adapter usually only adds a small number of parameters, which are specifically used to adjust the attention mechanism (self-attention and interactive attention) of the UNet network. Compared with directly training the entire UNet network to be trained, this method greatly reduces the number of parameters that need to be trained; during the training process, the LoRA adapter can improve the adaptability of the UNet network to different control conditions by adjusting the feature representation of the UNet network.

[0067] For ease of understanding, Figure 5 is a schematic diagram of an image generation model shown in this application, such as Figure 5 As shown, the image generation model is generated by a first preprocessing network, a second preprocessing network, a target UNet network and a decoder.

[0068] The first preprocessing network includes a depth map estimation unit, a first encoder, a second encoder, a noise adding unit and a first splicing unit.

[0069] Among them, after the sample image is input into the first preprocessing network, the first preprocessing network estimates the depth map of the sample image based on the depth map estimation unit to obtain the depth map of the sample image; then the depth map is encoded based on the first encoder to obtain the first latent space encoding, and the first latent space encoding is used as a control condition. It is not difficult to understand that the depth map provides geometric structure information of objects in the image, reflects the relative distance and position between different objects in the scene, and inputting the first latent space encoding corresponding to the depth map as a control condition can ensure that the image finally generated by the image generation model is consistent with the original image in terms of spatial structure and perspective.

[0070] Furthermore, the first preprocessing network encodes the sample image based on the second encoder to obtain a second latent space code; adds noise to the second latent space code based on the noise adding unit to obtain a third latent space code; and splices the third latent space code and the first latent space code based on the first splicing unit to obtain a target latent space code.

[0071] Among them, the first latent space encoding captures the geometric and spatial structural information of the image by encoding the depth map. The depth map itself contains the three-dimensional structure of the scene, and provides strong support for the position, shape, size and other information of the objects in the image; the second latent space encoding comes from the image itself, and contains the visual features of the image such as texture, color, and background. The two represent the spatial information and visual information of the image respectively. Combining these two types of information and splicing them together, the target latent space encoding obtained can simultaneously contain the geometric (depth) information and visual (texture, contour, etc.) information of the image. This combination improves the model's comprehensive understanding and generation capabilities of the image. Adding noise to the second latent space encoding through the noise adding unit can increase the robustness of the model.

[0072] The second preprocessing network includes a text encoder for converting sample prompt words into text vectors.

[0073] After obtaining the target latent space encoding, text vector and control conditions, the target latent space encoding, text vector and control conditions will be input into the UNet network to be trained, and the UNet network will be trained in combination with the noise distribution output by the UNet network to obtain the trained target UNet network. Among them, the control conditions are directly input into each adapter module of the UNet network to be trained. For the relationship between the UNet network to be trained and the adapter module, see the above detailed introduction.

[0074] Among them, the decoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

[0075] Figure 6is a schematic diagram of an exemplary implementation of a method for constructing an image generation model shown in the present application, such as Figure 6 As shown, the method for constructing the image generation model includes the following steps:

[0076] S601, determining a target Transformer module from candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network.

[0077] S602, coupling the adapter module between the self-attention module included in the target Transformer module and the upper layer of the self-attention module.

[0078] S603, coupling the pre-designed LoRa adapter to the self-attention module and the interactive attention module contained in the target Transformer module to obtain the UNet network to be trained.

[0079] Regarding the specific implementation of steps S601 to S603, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0080] S604, obtaining a sample image and a sample prompt word corresponding to the sample image.

[0081] S605, processing the sample image based on the first preprocessing network to obtain control conditions and target latent space encoding corresponding to the sample image.

[0082] S606: Convert the sample prompt word into a text vector based on the second preprocessing network.

[0083] S607, taking the target latent space encoding, the text vector and the control condition as the input of the UNet network to be trained, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network.

[0084] S608, constructing an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

[0085] Regarding the specific implementation of steps S604 to S608, reference may be made to the specific introduction of the relevant parts in the above embodiments, which will not be elaborated here.

[0086] The embodiment of the present application improves the original initial UNet network, couples in an adapter module and a LORA adapter, and uses the control conditions obtained based on the sample image as input information throughout the UNet network, so that a better target UNet network is obtained after training. The image generation model constructed based on the target UNet network can significantly improve the quality of the image finally generated by the image generation model, so that the finally generated image can highly represent the user features of the image input by the user.

[0087] Figure 7 is a schematic diagram of an exemplary implementation of an image generation method shown in the present application, such as Figure 7 As shown, the image generation method comprises the following steps:

[0088] S701, obtaining an initial image and prompt word input by a user.

[0089] When the user is using it online, the initial image and prompt words entered by the user are obtained.

[0090] S702, inputting the initial image into the first preprocessing network of the image generation model, inputting the prompt word into the second preprocessing network of the image generation model, and obtaining the target image output by the image generation model.

[0091] The image generation model is generated by using the image generation model construction method described in the above embodiment.

[0092] In an embodiment of the present application, when the user uses it online, the initial image and prompt words input by the user are input into the image generation model generated above. The target image output by the image generation model can be highly consistent with the initial image input by the user in terms of geometric information and visual information. At the same time, based on the prompt words, the target image output by the image generation model can be stylized, which better meets the needs of users and improves user satisfaction.

[0093] Figure 8 is a schematic diagram of a construction device of an image generation model shown in the present application, such as Figure 8 As shown, the image generation model construction device 800 includes an acquisition module 801, a first processing module 802, a second processing module 803, a training module 804 and a generation module 805, wherein:

[0094] The acquisition module 801 is used to acquire a sample image and a sample prompt word corresponding to the sample image.

[0095] The first processing module 802 is used to process the sample image based on the first preprocessing network to obtain the control condition and target latent space encoding corresponding to the sample image.

[0096] The second processing module 803 is used to convert the sample prompt word into a text vector based on the second preprocessing network.

[0097] The training module 804 is used to take the target latent space encoding, text vector and control condition as the input of the UNet network to be trained, and train the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network.

[0098] The generation module 805 is used to construct an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

[0099] In this device, by passing the control conditions as input information through the target UNet network, a better target UNet network is obtained after training. The image generation model constructed based on the target UNet network can significantly improve the quality of the image finally generated by the image generation model, so that the finally generated image can highly represent the user characteristics of the image input by the user.

[0100] Furthermore, the first preprocessing network includes a depth map estimation unit and a first encoder, and the first processing module 802 is also used to: perform depth map estimation on the sample image based on the depth map estimation unit to obtain a depth map of the sample image; encode the depth map based on the first encoder to obtain a first latent space encoding, and use the first latent space encoding as a control condition.

[0101] Furthermore, the first preprocessing network also includes a second encoder, a noise adding unit and a first splicing unit. The first processing module 802 is also used to: encode the sample image based on the second encoder to obtain a second latent space code; add noise to the second latent space code based on the noise adding unit to obtain a third latent space code; splice the third latent space code and the first latent space code based on the first splicing unit to obtain a target latent space code.

[0102] Furthermore, the image generation model construction device 800 also includes a UNet network construction module, which is used to: couple a pre-designed adapter module to the initial UNet network to obtain a UNet network to be trained; wherein the input of the adapter module includes control conditions, and the adapter module is used to integrate the control conditions into the UNet network.

[0103] Furthermore, the UNet network construction module is also used to: determine the target Transformer module from the candidate Transformer modules contained in the downsampling layer, upsampling layer and intermediate layer of the initial UNet network; couple the adapter module between the self-attention module contained in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

[0104] Furthermore, in the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated as the input of the self-attention module.

[0105] Furthermore, the adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, wherein: the resolution alignment unit includes multiple convolution kernels, which are used to align the spatial resolution of the control condition with the output of the previous layer; the second splicing unit is used to splice the control condition after the spatial resolution alignment with the output of the previous layer to obtain the splicing feature; the feature aggregation unit includes multiple residual blocks, which are used to fuse the splicing features to obtain the fused feature; the spatial attention unit is used to generate the weighting parameter corresponding to the fused feature based on the splicing feature; the multiplication unit is used to multiply the fused feature and the weighting parameter, wherein the result of the multiplication is used as the output of the adapter module.

[0106] Furthermore, the UNet network building module is also used to couple the pre-designed LoRa adapter to the self-attention module and the interactive attention module contained in the target Transformer module.

[0107] Furthermore, the decoder is used to convert the denoised output of the target UNet network after multiple time steps into the final generated image.

[0108] Fig. 9 is a schematic diagram of an image generating device shown in the present application, such as Fig. 9 As shown, the image generating device 900 comprises:

[0109] The acquisition module 901 is used to acquire the initial image and prompt word input by the user.

[0110] The generation module 902 is used to input the initial image into the first preprocessing network of the image generation model, input the prompt word into the second preprocessing network of the image generation model, and obtain the target image output by the image generation model.

[0111] The image generation model is generated by using the image generation model construction device introduced in the above embodiment.

[0112] When the device is used online by a user, the initial image and prompt words input by the user are input into the above-mentioned generated image generation model. The target image output by the image generation model can be highly consistent with the initial image input by the user in terms of geometric information and visual information. At the same time, based on the prompt words, the target image output by the image generation model can be stylized, which better meets the needs of the user and improves user satisfaction.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0114] Fig.10 A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0115] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0116] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the construction method of the image generation model and the image generation method. For example, in some embodiments, the construction method of the image generation model and the image generation method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the construction method of the image generation model and the image generation method described above may be executed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the image generation model construction method and the image generation method in any other appropriate manner (for example, by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for constructing an image generation model, wherein: include: Obtaining a sample image and a sample prompt word corresponding to the sample image; Processing the sample image based on a first preprocessing network to obtain a control condition and a target latent space encoding corresponding to the sample image; Converting the sample prompt word into a text vector based on a second preprocessing network; The target latent space encoding, the text vector and the control condition are used as inputs of a UNet network to be trained, and the UNet network is trained in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; An image generation model is constructed based on the first preprocessing network, the second preprocessing network, the target UNet network and the decoder.

2. The method according to claim 1, wherein: The first preprocessing network includes a depth map estimation unit and a first encoder, and the processing of the sample image based on the first preprocessing network to obtain a control condition corresponding to the sample image includes: Performing depth map estimation on the sample image based on the depth map estimation unit to obtain a depth map of the sample image; The depth map is encoded based on the first encoder to obtain a first latent space encoding, and the first latent space encoding is used as the control condition.

3. The method according to claim 2, wherein: The first preprocessing network further includes a second encoder, a noise adding unit and a first splicing unit, and the method for acquiring the target latent space encoding includes: Encoding the sample image based on the second encoder to obtain a second latent space code; Adding noise to the second latent space code based on the noise adding unit to obtain a third latent space code; The third latent space code and the first latent space code are concatenated based on the first concatenation unit to obtain the target latent space code.

4. The method according to any one of claims 1 to 3, wherein: The method for constructing the UNet network to be trained includes: The pre-designed adapter module is coupled to the initial UNet network to obtain the UNet network to be trained; The input of the adapter module includes the control condition, and the adapter module is used to integrate the control condition into the UNet network.

5. The method according to claim 4, wherein: The pre-designed adapter module is coupled to the initial UNet network to obtain the UNet network to be trained, including: Determine a target Transformer module from the candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network; The adapter module is coupled between the self-attention module included in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

6. The method according to claim 5, wherein: In the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated and used as the input of the self-attention module.

7. The method according to claim 6, wherein: The adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, wherein: The resolution alignment unit includes a plurality of convolution kernels, which are used to align the spatial resolution of the control condition with the output of the previous layer; The second splicing unit is used to splice the control condition after the spatial resolution is aligned with the output of the previous layer to obtain a splicing feature; The feature aggregation unit includes a plurality of residual blocks, which are used to fuse the spliced ​​features to obtain fused features; The spatial attention unit is used to generate a weighting parameter corresponding to the fusion feature based on the splicing feature; The multiplication unit is used to multiply the fused feature and the weighted parameter, wherein the multiplication result is used as the output of the adapter module.

8. The method according to claim 7, wherein: The method for constructing the UNet network to be trained further includes: A pre-designed LoRa adapter is coupled to the self-attention module and the interactive attention module included in the target Transformer module.

9. The method according to claim 1, wherein: The decoder is used to convert the denoised output of the target UNet network after multiple time steps into a final generated image.

10. A method for generating an image, wherein: include: Get the initial image and prompt word input by the user; Inputting the initial image into a first preprocessing network of an image generation model, inputting the prompt word into a second preprocessing network of the image generation model, and obtaining a target image output by the image generation model; Wherein, the image generation model is generated using the method described in any one of claims 1-9.

11. A device for constructing an image generation model, wherein: include: An acquisition module, used to acquire a sample image and a sample prompt word corresponding to the sample image; A first processing module, configured to process the sample image based on a first preprocessing network to obtain a control condition and a target latent space encoding corresponding to the sample image; A second processing module, configured to convert the sample prompt word into a text vector based on a second preprocessing network; A training module, used for taking the target latent space encoding, the text vector and the control condition as inputs of a UNet network to be trained, and training the UNet network in combination with the noise distribution output by the UNet network to obtain a trained target UNet network; A generation module is used to build an image generation model based on the first preprocessing network, the second preprocessing network, the target UNet network and a decoder.

12. The device according to claim 11, wherein: The first preprocessing network includes a depth map estimation unit and a first encoder, and the first processing module is further used for: Performing depth map estimation on the sample image based on the depth map estimation unit to obtain a depth map of the sample image; The depth map is encoded based on the first encoder to obtain a first latent space encoding, and the first latent space encoding is used as the control condition.

13. The device according to claim 12, wherein: The first preprocessing network further includes a second encoder, a noise adding unit and a first splicing unit, and the first processing module is further used for: Encoding the sample image based on the second encoder to obtain a second latent space code; Adding noise to the second latent space code based on the noise adding unit to obtain a third latent space code; The third latent space code and the first latent space code are concatenated based on the first concatenation unit to obtain the target latent space code.

14. The device according to any one of claims 11 to 13, wherein: The device also includes a UNet network construction module, and the UNet network construction module is used to: The pre-designed adapter module is coupled to the initial UNet network to obtain the UNet network to be trained; The input of the adapter module includes the control condition, and the adapter module is used to integrate the control condition into the UNet network.

15. The device according to claim 14, wherein: The UNet network building module is also used to: Determine a target Transformer module from the candidate Transformer modules included in the downsampling layer, upsampling layer, and intermediate layer of the initial UNet network; The adapter module is coupled between the self-attention module included in the target Transformer module and the upper layer of the self-attention module to obtain the UNet network to be trained.

16. The device according to claim 15, wherein: In the UNet network to be trained, the input of the adapter module is the output of the previous layer and the control condition; the output of the adapter module and the output of the previous layer are concatenated and used as the input of the self-attention module.

17. The device according to claim 16, wherein: The adapter module includes a resolution alignment unit, a second splicing unit, a feature aggregation unit, a spatial attention unit and a multiplication unit, wherein: The resolution alignment unit includes a plurality of convolution kernels, which are used to align the spatial resolution of the control condition with the output of the previous layer; The second splicing unit is used to splice the control condition after the spatial resolution is aligned with the output of the previous layer to obtain a splicing feature; The feature aggregation unit includes a plurality of residual blocks, which are used to fuse the spliced ​​features to obtain fused features; The spatial attention unit is used to generate a weighting parameter corresponding to the fusion feature based on the splicing feature; The multiplication unit is used to multiply the fused feature and the weighted parameter, wherein the multiplication result is used as the output of the adapter module.

18. The device according to claim 17, wherein: The UNet network building module is also used to: A pre-designed LoRa adapter is coupled to the self-attention module and the interactive attention module included in the target Transformer module.

19. The device according to claim 11, wherein: The decoder is used to convert the denoised output of the target UNet network after multiple time steps into a final generated image.

20. An image generating device, wherein: include: An acquisition module is used to acquire the initial image and prompt words input by the user; A generation module, used for inputting the initial image into a first preprocessing network of an image generation model, inputting the prompt word into a second preprocessing network of the image generation model, and obtaining a target image output by the image generation model; Wherein, the image generation model is generated by using the device described in any one of claims 11-19.

21. An electronic device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9 or claim 10.

22. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9 or claim 10.

23. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9 or claim 10.

Images