Image generation method and apparatus, device and medium

Abstract

The embodiments of the present disclosure relate to an image generation method and apparatus, a device and a medium. The method comprises: acquiring a target text; and, on the basis of the target text, using a preset target network model to generate a multi-channel target image, wherein the multi-channel target image has a color channel and an additional channel, the additional channel corresponds to additional information excepting color information, the target network model comprises a target diffusion network, and input channels of the target diffusion network comprise a first-type input channel corresponding to the color channel and a second-type input channel corresponding to the additional channel.

Description

Image generation methods, apparatus, devices and media

[0001] Cross-references to related applications

[0002] This application claims priority to Chinese Patent Application No. 202410520318.6, filed on April 26, 2024, entitled “Image Generation Method, Apparatus, Device and Medium”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of computer technology, and in particular to an image generation method, apparatus, device, and medium. Background Technology

[0004] Image generation technology has always been a research hotspot in scenarios requiring image processing, such as digital image processing and computer vision. Summary of the Invention

[0005] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides an image generation method, apparatus, device and medium.

[0006] This disclosure provides an image generation method, the method comprising: acquiring target text; generating a multi-channel target image based on the target text using a preset target network model; wherein the multi-channel target image has a color channel and an additional channel, the additional channel corresponding to additional information other than color information; the target network model includes a target diffusion network, the input channels of the target diffusion network including a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channel.

[0007] This disclosure also provides an image generation apparatus, comprising: a text acquisition module for acquiring target text; and an image generation module for generating a multi-channel target image based on the target text using a preset target network model; wherein the multi-channel target image has a color channel and an additional channel, the additional channel corresponding to additional information other than color information; the target network model includes a target diffusion network, and the input channels of the target diffusion network include a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channel.

[0008] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image generation method provided in this disclosure.

[0009] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image generation method provided in this disclosure.

[0010] The technical solution provided in this disclosure can generate a multi-channel target image with color channels and additional channels based on target text using a preset target network model. The target network model includes a target diffusion network, and the input channels of the target diffusion network include a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channels.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0013] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 is a schematic flowchart of an image generation method provided in an embodiment of this disclosure;

[0015] Figure 2 is a schematic diagram of the structure of a target network model provided in an embodiment of this disclosure;

[0016] Figure 3 is a schematic diagram of an RGBA image generation process provided in an embodiment of this disclosure;

[0017] Figure 4 is a schematic diagram of a network training process provided in an embodiment of this disclosure;

[0018] Figure 5 is a schematic diagram of the structure of a target network model provided in an embodiment of this disclosure;

[0019] Figure 6 is a schematic diagram of an RGBA image generation process provided in an embodiment of this disclosure;

[0020] Figure 7 is a schematic diagram of a network training process provided in an embodiment of this disclosure;

[0021] Figure 8 is a schematic diagram of an image generation apparatus provided in an embodiment of this disclosure;

[0022] Figure 9 is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0023] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0024] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0025] In related technologies, generating images with additional information besides color information is cumbersome and complex, and the results are unsatisfactory. Taking transparency information as an example, to generate an RGBA image with both RGB and transparency channels, related technologies need to first generate an RGB image, and then separate the foreground and background to obtain the corresponding transparency information alpha channel (hereinafter referred to as the transparency channel). However, this method is cumbersome, and the generated transparency information has strong limitations, basically only achieving a fully transparent or fully opaque effect, and cannot be applied to scenes that require complex transparency changes such as glass effects or smoke effects. The RGBA images generated by related technologies are difficult to meet the requirements. The inventors have found that although related technologies have achieved significant results in generating RGB images, they still have certain limitations in generating images with additional information besides color information, such as the fact that the methods used are usually cumbersome and complex, and the results are unsatisfactory.

[0026] To improve one or more of the above problems, this disclosure provides an image generation method, apparatus, device, and medium, which are described in detail below.

[0027] Figure 1 is a flowchart illustrating an image generation method according to an embodiment of this disclosure. This method can be executed by an image generation device, which can be implemented using software and / or hardware and is generally integrated into an electronic device. As shown in Figure 1, the method mainly includes the following steps S102 to S104:

[0028] Step S102: Obtain the target text. This embodiment of the disclosure does not limit the content of the target text; for example, the target text can be used to prompt information such as the content of the image to be generated.

[0029] Step S104: Based on the target text, generate a multi-channel target image using a preset target network model; wherein, the multi-channel target image has a color channel and an additional channel, the additional channel corresponding to additional information other than color information; the target network model includes a target diffusion network, and the input channels of the target diffusion network include a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channel.

[0030] The aforementioned target network model can be a neural network model. This target network model needs to include a target diffusion network with improved channels. It is understood that existing diffusion networks are basically only used to generate RGB images, so their input channels only correspond to color channels. However, in order to directly output a multi-channel target image with both color channels and additional channels using the diffusion network, the input channels of the target diffusion network can be improved by adding a second type of input channel corresponding to the additional channels. This allows the target diffusion network to generate additional information for the multi-channel target image based on the information input from the second type of input channel. This disclosure does not limit the additional information; any image information other than color information can be used as additional information. For example, additional information includes, but is not limited to, one or more of transparency information, depth information, and thermal information. The channel corresponding to transparency information can be called an alpha channel or transparency channel; the channel corresponding to depth information can be called a depth channel; and the channel corresponding to thermal information can be called a thermal channel. For example, the aforementioned color channels may include an R channel (Red channel), a G channel (Green channel), and a B channel (Blue channel). If the additional channel is a transparency channel, then the multi-channel target image is a four-channel RGBA image.

[0031] The above method can be used to efficiently generate multi-channel target images by directly utilizing the improved network model of the input channel, so that it presents both color information and additional information. The above method of generating multi-channel images by leveraging the strong network processing capabilities is simpler and more efficient, and can better guarantee the generation effect of multi-channel images.

[0032] The target network model includes a target diffusion network and may also include other networks. The structure of the target network model is not limited here; for example, the target network model can be a single model or a combination of multiple models. For ease of understanding, this disclosure provides exemplary implementations for generating multi-channel target channels using a target network model, which are described below:

[0033] The target network model includes a first network model and a second network model. Based on this, step S104 above, that is, the step of generating a multi-channel target image based on the target text using a preset target network model, can be performed as follows: steps a to c:

[0034] Step a: Based on the target text, generate a first image with only color channels using a first network model. The color channels can be RGB three channels. The aforementioned first network model can be implemented by fine-tuning and pre-training based on a preset diffusion model. For example, the preset diffusion model can be trained using RGB sample images, so that the trained first network model can generate an RGB image that matches the input text based on a cross-attention mechanism, such as generating an RGB image with a solid color background.

[0035] Step b: Based on the first image, a second image with only additional channels is generated using a second network model. The first image is an RGB image. The second network model can further predict additional information based on the RGB image, thus obtaining a second image with only additional channels. Taking the additional information as transparency information as an example, the second image is an image with only a transparency channel, also known as an alpha image. In practical applications, the structure of the second network model can be improved, such as changing the input channels of the diffusion network in the second network model, and training the second network model using alpha sample images, thereby enabling the second network model to directly generate alpha images.

[0036] Step c: Merge the first image and the second image to obtain a multi-channel target image.

[0037] For ease of understanding, in some specific implementation examples, a schematic diagram of a target network model can be referenced in Figure 2. The first network model includes a first encoder, a first diffusion network, and a first decoder; the second network model includes a second encoder, a second diffusion network, and a second decoder; wherein, the second diffusion network is used as the target diffusion network. In practical applications, the structures of the first and second network models are the same, but the number of input channels of the second diffusion network can be improved by adding input channels corresponding to additional channels in addition to the original input channels corresponding to the color channels.

[0038] This disclosure further provides a specific implementation example of step b above, namely, generating a second image with only additional channels based on the first image using a second network model, which can be performed with reference to the following steps b1 to b3:

[0039] Step b1: Based on the first image, generate a first latent representation corresponding to the color channels of the first image using a second encoder. For example, the first image is an RGB image, that is, the input of the second encoder is three-channel data. The second encoder can be implemented using an existing encoder. Typically, the second encoder can encode the input data and output four-channel data. That is, the first latent representation output by the second encoder is four-channel data.

[0040] Step b2: Based on the first latent representation and the preset first noise, a second latent representation corresponding to the additional channels of the second image is generated using a second diffusion network. The first noise can be randomly set Gaussian noise. In practical applications, the first type of input channels of the second diffusion network are used to input the first latent representation; the second type of input channels of the second diffusion network are used to input the first noise; the number of channels in the second type of input channels is equal to the number of channels in the first type of input channels, that is, the number of channels in the first noise is the same as the number of channels in the first latent representation. For example, if both the first latent representation and the first noise have 4 channels, then the second diffusion network has a total of 8 input channels. It is understood that the number of input channels in existing diffusion networks is usually 4. This embodiment can improve upon this by expanding the number of input channels of the second diffusion network from 4 to 8, where 4 channels are used to input the first latent representation and the other 4 channels are used to input Gaussian noise. Using the second diffusion network, noise prediction can be performed based on its input information through a diffusion sampling process to obtain the second latent representation based on the predicted noise. The above diffusion sampling process can be implemented with reference to relevant technologies and can be the same as the diffusion sampling process used in existing diffusion networks, so it will not be described in detail here. The output channels of the second diffusion network can be consistent with the output channels of the existing diffusion network. For example, the second diffusion network has 4 output channels, and the second potential is represented as four-channel data.

[0041] Step b3 involves generating a second image with only the additional channels using the second decoder, based on the second latent representation. In some specific implementation examples, step b3 can be performed with reference to steps b3.1 and b3.2 as follows:

[0042] Step b3.1: Based on the second latent representation, an intermediate image is generated using a second decoder; wherein the intermediate image has multiple additional channels corresponding to the same type of additional information. The second decoder can be implemented using an existing decoder. Typically, the decoder can decode the input data and output three-channel data. That is, the intermediate image output by the second decoder has three additional channels, and the three additional channels correspond to the same type of additional information. Taking the additional channel as the transparency channel as an example, the intermediate image is an image with three transparency channels, which can also be called a three-channel alpha image.

[0043] Step b3.2: Generate a second image based on the intermediate image; wherein the second image has an additional channel corresponding to the same additional information. Due to the characteristics of the decoder, it can only output an intermediate image with multiple channels, but the intermediate image does not meet the requirements. Therefore, it is necessary to convert the intermediate image into an image with only one additional channel, such as converting an image with three transparency channels into an image with one transparency channel. For example, step b3.2 can be performed in the following manner (1) or (2):

[0044] Method (1) involves averaging multiple additional channels corresponding to the same additional information in the intermediate image to obtain a second image. Specifically, the information corresponding to multiple additional channels can be averaged, and a second image with one additional channel can be obtained based on the averaging result. The averaging result is the information corresponding to that additional channel.

[0045] Method (2) involves selecting one additional channel from multiple additional channels corresponding to the same additional information in the intermediate image to obtain the second image. This embodiment does not limit the selection method. Considering that the information corresponding to the three additional channels output by the decoder is largely consistent, in some specific examples, one additional channel can be randomly selected, and the information of the other two additional channels can be discarded to obtain the second image. This method can more efficiently convert the intermediate image into a single-channel second image, such as converting a three-channel alpha image into the desired single-channel alpha image.

[0046] For ease of understanding, taking transparency information as an example, refer to Figure 3 for a schematic diagram of an RGBA image generation process. This provides the RGBA image generation process of the target network model during the inference stage. The first encoder in the first network model can be one or more. The first network model illustrates a first encoder 1 (which can be a text encoder) used for input text. In addition, a first encoder 2 is provided, which can be used to input other information that needs to be encoded. The first diffusion network in the first network model can be a Unet network, which can process the text features and other information output by the first encoder 1 based on a cross-attention mechanism. Noise prediction is performed through diffusion sampling to obtain the RGB four-channel latent representation. Based on this, the first solution... The encoder obtains an RGB three-channel image through decoding. This RGB three-channel image is then input to the second encoder in the second network model for encoding, resulting in a four-channel latent representation of the RGB image. Four channels of Gaussian noise are added and input to a second diffusion network with eight input channels (Unet can also be used). Based on the input information, the second diffusion network performs noise prediction through diffusion sampling to obtain the four-channel latent representation of the alpha image. The second decoder then obtains a three-channel alpha image through decoding, which is then converted into the desired single-channel alpha image. Finally, the single-channel alpha image is merged with the three-channel RGB image to obtain the RGBA image. This method provides a convenient and quick way to obtain RGBA images. Furthermore, the transparency information generated by the diffusion network does not suffer from the limitations of foreground / background segmentation methods used in related technologies. For example, it is not limited to completely transparent or completely opaque; it can flexibly take values ​​between 0 and 1, generating complex transparency information. Ultimately, the resulting RGBA image can adapt to more scenarios and better meet user needs.

[0047] Furthermore, this disclosure provides a method for obtaining the second diffusion network. Exemplarily, the second diffusion network is obtained through training via the following steps A to D:

[0048] Step A: Obtain a first multi-channel sample image and perform a channel separation operation on the first multi-channel sample image to obtain a first sample image with only color channels and a second sample image with only additional channels. For example, taking the additional information as transparency information, the first multi-channel sample image can be an RGBA sample image. The RGB channels and transparency channels of the RGBA sample image can be separated to obtain an RGB sample image (i.e., the aforementioned first sample image) and an alpha sample image (i.e., the aforementioned second sample image).

[0049] Step B involves obtaining the third latent representation corresponding to the color channels of the first sample image and the fourth latent representation corresponding to the additional channels of the second sample image. For example, step B can be performed with reference to steps B1 through B3 as follows:

[0050] Step B1 involves inputting the first sample image into the second encoder to obtain the third latent representation corresponding to the color channels of the first sample image. As mentioned earlier, the third latent representation output by the second encoder is four-channel data.

[0051] Step B2 involves performing channel duplication on the second sample image to ensure that the number of additional channels in the duplicated second sample image matches the number of input channels in the second encoder. As mentioned earlier, the second encoder can be implemented using an existing encoder. Typically, the second encoder has three input channels. Therefore, it is necessary to duplicate the channels of the single-channel second sample image (such as a single-channel alpha image) three times to obtain a three-channel alpha image, which can then be input into the encoder for normal processing.

[0052] Step B3 involves inputting the second sample image, after channel duplication processing, into the second encoder to obtain the fourth latent representation corresponding to the additional channels of the second sample image. The fourth latent representation has the same number of channels as the third latent representation, such as four-channel data.

[0053] Step C: Based on the third latent representation and the fourth latent representation after adding noise, the predicted latent representation corresponding to the additional channels of the second sample image is obtained using a preset initial network. Specifically, the first type of input channel of the preset initial network is used to input the third latent representation, and the second type of input channel of the preset initial network is used to input the fourth latent representation after adding noise.

[0054] Step D involves training a pre-set initial network based on the difference between the fourth latent representation and the predicted latent representation, to obtain a second diffusion network. It is understood that the pre-set initial network and the second diffusion network have the same structure. Training is considered complete when the parameters of the pre-set initial network are optimized to achieve the desired result, thus obtaining the second diffusion network. The pre-set initial network and the second diffusion network have the same number of input channels, but the information used for input channels differs. For example, the second type of input channel of the pre-set initial network is used to input the fourth latent representation after adding noise, while the second type of input channel of the second diffusion network is used to input Gaussian noise.

[0055] For ease of understanding, taking transparency information as an example, refer to Figure 4 for a schematic diagram of a network training process. This diagram primarily illustrates how a preset initial network is trained to obtain the aforementioned second diffusion network. Figure 4 clearly shows that during the training phase, the single-channel alpha sample image needs to undergo channel duplication to obtain a three-channel alpha sample image. This allows the second encoder to encode the three-channel alpha sample image, resulting in a four-channel latent representation corresponding to the alpha sample image. Additionally, the three-channel RGB sample image is also input to the second encoder, yielding a four-channel latent representation corresponding to the RGB sample image. Furthermore, Figure 4 clearly shows that the input to the preset initial network includes the four-channel latent representation corresponding to the RGB sample image and the four-channel latent representation corresponding to the alpha sample image after adding noise. This differs from the input to the second diffusion network (trained based on the preset initial network) shown in Figure 3. The preset initial network can perform noise prediction to predict the latent representation corresponding to the alpha sample image. The loss is calculated based on the difference between this latent representation and the true latent representation corresponding to the alpha sample image, allowing adjustment of the preset initial network's parameters. This enables training of the preset initial network, resulting in a second diffusion network that outputs as expected after training.

[0056] Referring to the schematic diagram of a target network model shown in Figure 5, the target network model includes a third encoder, a third diffusion network, and a third decoder; wherein, the third diffusion network is used as the target diffusion network. Based on this, the above step S104, that is, the step of generating a multi-channel target image based on the target text using a preset target network model, can be performed as follows: Steps 1 to 3:

[0057] Step 1: Based on the target text, generate the text features corresponding to the target text using a third encoder. The third encoder can also be implemented with reference to existing encoders, and will not be described in detail here.

[0058] Step 2: Based on text features and preset second noise, use the third diffusion network to generate the fifth latent representation corresponding to the color channels of the multi-channel target image and the sixth latent representation corresponding to the additional channels of the multi-channel target image.

[0059] The second noise can be initialized Gaussian noise. In practical applications, both the first and second input channels of the third diffusion network are used to input the second noise. Assuming the third diffusion network has four first and four second input channels respectively, this is equivalent to having eight input channels, and the second noise can be eight-channel noise. The third diffusion network can perform cross-attention processing based on the second noise and text features to predict the noise, and obtain the fifth and sixth latent representations based on the noise prediction results. The number of output channels of the third diffusion network is the same as the number of input channels, such as eight, meaning that an eight-channel latent representation can be obtained through the third diffusion network. Specifically, this can be understood as a four-channel fifth latent representation and a four-channel sixth latent representation.

[0060] Step 3: Based on the fifth and sixth latent representations, generate a multi-channel target image using the third decoder. For example, step 3 can be performed with reference to steps 3.1 and 3.3 below:

[0061] Step 3.1: Based on the fifth latent representation and the sixth latent representation, use the third decoder to generate a third image with only color channels and a fourth image with only additional channels; as mentioned above, the decoder output is usually three-channel data, so the fourth image has multiple additional channels corresponding to the same additional information, such as the fourth image being a three-channel alpha image.

[0062] Step 3.2: Generate a fifth image based on the fourth image; wherein the fifth image has an additional channel corresponding to the same additional information. As mentioned earlier, due to the characteristics of the decoder, it can only output a fourth image with multiple channels, but the fourth image does not meet the requirements. Therefore, it is necessary to convert the fourth image into a fifth image with only one additional channel, such as converting an image with three transparency channels into an image with one transparency channel. Specifically, this can be done by averaging multiple additional channels or by selecting one additional channel from multiple additional channels. Refer to the relevant content of step b3.2 in the aforementioned embodiment, which will not be repeated here.

[0063] Step 3.3: Merge the third image and the fifth image to obtain a multi-channel target image.

[0064] For ease of understanding, taking transparency information as an example, the generation process of an RGBA image shown in Figure 6 illustrates the generation process of the RGBA image in the inference stage of the target network model. Figure 6 shows the first encoder 1 (which can be a text encoder) used for input text and the third diffusion network. In addition, it can also include encoders for encoding other information, which is not limited here and is not shown in Figure 6. The third diffusion network can be a Unet network, whose input is eight-channel Gaussian noise. It can process the text features and Gaussian noise output by the first encoder 1 based on the cross-attention mechanism, and perform noise prediction through diffusion sampling to obtain the RGB four-channel latent representation and the alpha four-channel latent representation. On this basis, the third decoder can obtain the three-channel RGB image and the three-channel alpha image through decoding. The three-channel alpha image also needs to be converted into a single-channel alpha image and then merged with the three-channel RGB image to obtain the RGBA image.

[0065] Furthermore, this disclosure provides a method for obtaining the third diffusion network. Exemplarily, the third diffusion network is trained through the following steps one through four:

[0066] Step one: Acquire a second multi-channel sample image, and perform a channel separation operation on the second multi-channel sample image to obtain a third sample image with only color channels and a fourth sample image with only additional channels. The second multi-channel sample image may be the same as or different from the aforementioned first multi-channel sample image, and this is not restricted here.

[0067] Step two involves obtaining the seventh latent representation corresponding to the color channels of the third sample image and the eighth latent representation corresponding to the additional channels of the fourth sample image. In practical applications, the seventh and eighth latent representations can be obtained using a third encoder. The specific acquisition method can refer to the implementation of step B in the aforementioned embodiments. Here, the fourth sample image also needs to undergo channel duplication processing so that the third encoder can properly process the channel-duplicated third sample image to obtain the eighth latent representation. For example, the eighth latent representation is also four-channel data, and the seventh latent representation is also four-channel data.

[0068] Step 3: Based on the seventh and eighth latent representations with added noise, noise prediction is performed using the initial diffusion network. Specifically, the first type of input channel of the initial diffusion network is used to input the seventh latent representation with added noise, and the second type of input channel is used to input the eighth latent representation with added noise. For example, the initial diffusion network has a total of 8 output channels.

[0069] Step four involves training the initial diffusion network based on the noise predicted by the initial diffusion network, the noise added to the seventh latent representation, and the noise added to the eighth latent representation, to obtain the third diffusion network. The loss can be calculated based on the difference between the noise predicted by the initial diffusion network for each channel and the noise received by the initial diffusion network for each channel, and training can be performed via backpropagation. It is understood that the preset diffusion network and the third diffusion network have the same structure. Training is considered complete when the parameters of the preset diffusion network are optimized until the expected results are achieved, resulting in the third diffusion network. The preset and third diffusion networks have the same number of input channels, but the information used for input channels differs. For example, the first type of input channel in the preset diffusion network is used to input the seventh latent representation after adding noise, and the second type of input channel is used to input the eighth latent representation after adding noise, while both the first and second type of input channels in the second diffusion network are used to input Gaussian noise.

[0070] For ease of understanding, taking transparency information as an example, refer to Figure 7, which illustrates a network training process. This diagram primarily shows how to train the initial diffusion network to obtain the aforementioned third diffusion network. The input data processing in Figure 7 is similar to that in Figure 4, such as the need to copy the alpha channel, which will not be elaborated further here. It should be noted that, unlike the third diffusion network in Figure 6 where all eight input channels are used for noise input, in Figure 7, the eight input channels of the initial diffusion network are used for: the four-channel latent representation of the RGB sample image after noise addition, and the four-channel latent representation of the alpha sample image after noise addition. The initial diffusion network can predict noise and directly calculate the loss based on the difference between the noise predicted by the initial diffusion network for each channel and the actual noise added to the input channels of the initial diffusion network. This allows for adjustment of the parameters of the initial diffusion network, enabling training. After training, a third diffusion network with the expected output can be obtained.

[0071] In some examples, multiple diffusion networks are involved, each focusing on a different stage of image generation. This minimizes the scope of modification for each network and reduces the disruption to the original network's generative capabilities. For instance, one network model might generate an RGB image, while another generates an alpha image. In other examples, a single diffusion network can simultaneously obtain both RGB and alpha data, enabling a faster and more convenient generation of RGBA images. Therefore, in practical applications, the target network model can be flexibly implemented according to requirements to generate multi-channel target images.

[0072] In summary, the image generation method provided in this disclosure can efficiently generate multi-channel target images by directly utilizing the improved network model of the input channel, so that it presents both color information and additional information. The above-mentioned method of generating multi-channel images by leveraging strong network processing capabilities is simpler and more efficient, and can better guarantee the generation effect of multi-channel images. For example, the transparency information of the image can be generated according to the requirements, and the image transparency can vary between 0 and 1, thereby meeting complex transparency change scenarios.

[0073] Corresponding to the aforementioned image generation method, Figure 8 is a schematic diagram of an image generation device provided in an embodiment of this disclosure. This device can be implemented by software and / or hardware, and is generally integrated into an electronic device. As shown in Figure 8, the image generation device includes:

[0074] Text acquisition module 802 is used to acquire target text;

[0075] Image generation module 804 is used to generate multi-channel target images based on target text using a preset target network model;

[0076] The multi-channel target image has color channels and additional channels, with the additional channels corresponding to additional information other than color information; the target network model includes a target diffusion network, and the input channels of the target diffusion network include a first type of input channel corresponding to the color channels and a second type of input channel corresponding to the additional channels.

[0077] The above-mentioned device can be used to efficiently generate multi-channel target images by directly utilizing the improved network model of the input channel, so that it presents both color information and additional information. The above-mentioned method of generating multi-channel images by leveraging the strong network processing capability is simpler and more efficient, and can better guarantee the generation effect of multi-channel images.

[0078] In some implementations, the additional information includes one or more of transparency information, depth information, and thermal information.

[0079] In some implementations, the target network model includes a first network model and a second network model; the image generation module 804 is specifically used to: generate a first image with only the color channels based on the target text using the first network model; generate a second image with only the additional channels based on the first image using the second network model; and merge the first image and the second image to obtain a multi-channel target image.

[0080] In some implementations, the first network model includes a first encoder, a first diffusion network, and a first decoder; the second network model includes a second encoder, a second diffusion network, and a second decoder; wherein the second diffusion network is used as the target diffusion network.

[0081] In some embodiments, the image generation module 804 is specifically used to: generate a first latent representation corresponding to the color channels of the first image using the second encoder based on the first image; generate a second latent representation corresponding to the additional channels of the second image using the second diffusion network based on the first latent representation and a preset first noise; and generate a second image having only the additional channels using the second decoder based on the second latent representation.

[0082] In some embodiments, the image generation module 804 is specifically used to: generate an intermediate image based on the second latent representation using the second decoder; wherein the intermediate image has multiple additional channels corresponding to the same additional information; and generate a second image based on the intermediate image; wherein the second image has one additional channel corresponding to the same additional information.

[0083] In some embodiments, the image generation module 804 is specifically used to: perform average processing on multiple additional channels corresponding to the same additional information of the intermediate image to obtain a second image; or, select one additional channel from the multiple additional channels corresponding to the same additional information of the intermediate image to obtain a second image.

[0084] In some implementations, a first type of input channel of the second diffusion network is used to input the first latent representation; a second type of input channel of the second diffusion network is used to input the first noise; and the number of channels of the second type of input channel is equal to the number of channels of the first type of input channel.

[0085] In some embodiments, the apparatus further includes a first training module for training the second diffusion network through the following steps: acquiring a first multi-channel sample image and performing a channel separation operation on the first multi-channel sample image to obtain a first sample image having only the color channel and a second sample image having only the additional channel; acquiring a third latent representation corresponding to the color channel of the first sample image and a fourth latent representation corresponding to the additional channel of the second sample image; based on the third latent representation and the fourth latent representation after adding noise, obtaining a predicted latent representation corresponding to the additional channel of the second sample image using a preset initial network; and training the preset initial network based on the difference between the fourth latent representation and the predicted latent representation to obtain the second diffusion network based on the trained preset initial network.

[0086] In some implementations, the first training module is specifically used to: input the first sample image into the second encoder to obtain a third latent representation corresponding to the color channels of the first sample image; perform channel duplication processing on the second sample image so that the number of additional channels of the second sample image after channel duplication processing is consistent with the number of input channels of the second encoder; and input the second sample image after channel duplication processing into the second encoder to obtain a fourth latent representation corresponding to the additional channels of the second sample image.

[0087] In some implementations, the first type of input channel of the preset initial network is used to input the third latent representation, and the second type of input channel of the preset initial network is used to input the fourth latent representation after adding noise.

[0088] In some implementations, the target network model includes a third encoder, a third diffusion network, and a third decoder; wherein the third diffusion network is used as the target diffusion network.

[0089] In some embodiments, the image generation module 804 is specifically used to: generate text features corresponding to the target text using the third encoder based on the target text; generate a fifth latent representation corresponding to the color channels of the multi-channel target image and a sixth latent representation corresponding to the additional channels of the multi-channel target image using the third diffusion network based on the text features and a preset second noise; and generate the multi-channel target image using the third decoder based on the fifth latent representation and the sixth latent representation.

[0090] In some implementations, both the first type of input channel and the second type of input channel of the third diffusion network are used to input the second noise.

[0091] In some implementations, the number of output channels of the third diffusion network is the same as the number of input channels of the third diffusion network.

[0092] In some embodiments, the image generation module 804 is specifically used to: generate a third image having only the color channel and a fourth image having only the additional channel based on the fifth latent representation and the sixth latent representation using the third decoder; wherein the fourth image has multiple additional channels corresponding to the same additional information; generate a fifth image based on the fourth image; wherein the fifth image has one additional channel corresponding to the same additional information; and merge the third image and the fifth image to obtain the multi-channel target image.

[0093] In some embodiments, the apparatus further includes a second training module for acquiring a second multi-channel sample image and performing a channel separation operation on the second multi-channel sample image to obtain a third sample image having only the color channel and a fourth sample image having only the additional channel; acquiring a seventh latent representation corresponding to the color channel of the third sample image and an eighth latent representation corresponding to the additional channel of the fourth sample image; performing noise prediction using an initial diffusion network based on the seventh latent representation with added noise and the eighth latent representation with added noise; and training the initial diffusion network based on the noise predicted by the initial diffusion network, the noise added to the seventh latent representation, and the noise added to the eighth latent representation to obtain a third diffusion network based on the trained initial diffusion network.

[0094] In some implementations, a first type of input channel of the initial diffusion network is used to input the seventh latent representation after adding noise, and a second type of input channel of the initial diffusion network is used to input the eighth latent representation after adding noise.

[0095] The image generation apparatus provided in this disclosure can execute the image generation method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0096] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device embodiments can be referred to the corresponding process in the method embodiments, and will not be repeated here.

[0097] This disclosure provides an electronic device, which includes: a storage device storing a computer program thereon; and a processing device for executing the computer program in the storage device to implement the steps of any method of this disclosure.

[0098] Referring now to FIG9, a schematic diagram of the structure of an electronic device 900 suitable for implementing embodiments of the present disclosure is shown. The terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG9 is merely an example and should not impose any limitation on the functionality and scope of use of embodiments of the present disclosure.

[0099] As shown in Figure 9, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0100] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although FIG9 shows an electronic device 900 with various devices, it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0101] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0102] In addition to the methods and devices described above, embodiments of this disclosure can also be computer program products, comprising computer program instructions that, when executed by a processor, cause the processor to perform the image processing methods provided in the embodiments of this disclosure. The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the image generation method provided in embodiments of this disclosure.

[0104] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0105] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image generation method of this disclosure.

[0106] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0107] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0108] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0109] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0110] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0111] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image generation method, comprising: Get the target text; Based on the target text, a multi-channel target image is generated using a preset target network model; The multi-channel target image has a color channel and an additional channel, the additional channel corresponding to additional information other than color information; the target network model includes a target diffusion network, the input channels of the target diffusion network include a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channel.

2. The method according to claim 1, wherein the additional information includes one or more of transparency information, depth information, and thermal information.

3. The method according to claim 1, wherein the target network model includes a first network model and a second network model; the step of generating a multi-channel target image based on the target text using a preset target network model includes: Based on the target text, a first image with only the color channels is generated using the first network model; Based on the first image, a second image with only the additional channels is generated using the second network model; The first image and the second image are merged to obtain a multi-channel target image.

4. The method according to claim 3, wherein the first network model includes a first encoder, a first diffusion network, and a first decoder; the second network model includes a second encoder, a second diffusion network, and a second decoder; wherein, The second diffusion network serves as the target diffusion network.

5. The method of claim 4, wherein generating a second image with only the additional channels based on the first image using the second network model comprises: Based on the first image, the second encoder is used to generate a first latent representation corresponding to the color channels of the first image; Based on the first latent representation and the preset first noise, the second diffusion network is used to generate a second latent representation corresponding to the additional channels of the second image; Based on the second latent representation, a second image with only the additional channels is generated using the second decoder.

6. The method of claim 5, wherein generating a second image having only the additional channel using the second decoder based on the second latent representation comprises: Based on the second latent representation, an intermediate image is generated using the second decoder; wherein the intermediate image has multiple additional channels corresponding to the same additional information; A second image is generated based on the intermediate image; wherein the second image has an additional channel corresponding to the same additional information.

7. The method of claim 6, wherein generating the second image based on the intermediate image comprises: The second image is obtained by averaging multiple additional channels corresponding to the same additional information in the intermediate image. or, The second image is obtained by selecting one additional channel from multiple additional channels corresponding to the same additional information in the intermediate image.

8. The method of claim 5, wherein a first type of input channel of the second diffusion network is used to input the first latent representation; a second type of input channel of the second diffusion network is used to input the first noise; and the number of channels of the second type of input channel is equal to the number of channels of the first type of input channel.

9. The method of claim 4, wherein the second diffusion network is trained by the following steps: A first multi-channel sample image is acquired, and a channel separation operation is performed on the first multi-channel sample image to obtain a first sample image with only the color channel and a second sample image with only the additional channel; Obtain the third latent representation corresponding to the color channel of the first sample image, and the fourth latent representation corresponding to the additional channel of the second sample image; Based on the third latent representation and the fourth latent representation after adding noise, the predicted latent representation corresponding to the additional channels of the second sample image is obtained using a preset initial network; Based on the difference between the fourth latent representation and the predicted latent representation, the preset initial network is trained to obtain the second diffusion network based on the trained preset initial network.

10. The method of claim 9, wherein obtaining the third latent representation corresponding to the color channels of the first sample image and the fourth latent representation corresponding to the additional channels of the second sample image comprises: The first sample image is input into the second encoder to obtain the third latent representation corresponding to the color channels of the first sample image; The second sample image is subjected to channel duplication processing so that the number of additional channels in the second sample image after channel duplication processing is consistent with the number of input channels of the second encoder; The second sample image, after channel duplication processing, is input into the second encoder to obtain the fourth latent representation corresponding to the additional channels of the second sample image.

11. The method of claim 9, wherein the first type of input channel of the preset initial network is used to input the third latent representation, and the second type of input channel of the preset initial network is used to input the fourth latent representation after adding noise.

12. The method according to claim 1, wherein the target network model comprises a third encoder, a third diffusion network, and a third decoder; wherein, The third diffusion network is used as the target diffusion network.

13. The method according to claim 12, wherein generating a multi-channel target image based on the target text using a preset target network model comprises: Based on the target text, the third encoder is used to generate text features corresponding to the target text; Based on the text features and the preset second noise, the third diffusion network is used to generate the fifth latent representation corresponding to the color channels of the multi-channel target image and the sixth latent representation corresponding to the additional channels of the multi-channel target image; The multi-channel target image is generated using the third decoder based on the fifth and sixth latent representations.

14. The method of claim 13, wherein both the first type of input channel and the second type of input channel of the third diffusion network are used to input the second noise.

15. The method of claim 13, wherein the number of output channels of the third diffusion network is the same as the number of input channels of the third diffusion network.

16. The method of claim 13, wherein generating the multi-channel target image using the third decoder based on the fifth latent representation and the sixth latent representation comprises: Based on the fifth latent representation and the sixth latent representation, the third decoder generates a third image having only the color channel and a fourth image having only the additional channels; wherein the fourth image has multiple additional channels corresponding to the same additional information; A fifth image is generated based on the fourth image; wherein the fifth image has an additional channel corresponding to the same additional information; The third image and the fifth image are merged to obtain the multi-channel target image.

17. The method of claim 13, wherein the third diffusion network is trained by the following steps: A second multi-channel sample image is acquired, and a channel separation operation is performed on the second multi-channel sample image to obtain a third sample image with only the color channel and a fourth sample image with only the additional channel. Obtain the seventh latent representation corresponding to the color channel of the third sample image, and the eighth latent representation corresponding to the additional channel of the fourth sample image; Noise prediction is performed using the initial diffusion network based on the seventh latent representation with added noise and the eighth latent representation with added noise. The initial diffusion network is trained based on the noise predicted by the initial diffusion network, the noise added to the seventh latent representation, and the noise added to the eighth latent representation, so as to obtain a third diffusion network based on the trained initial diffusion network.

18. The method of claim 17, wherein the first type of input channel of the initial diffusion network is used to input the seventh latent representation after adding noise, and the second type of input channel of the initial diffusion network is used to input the eighth latent representation after adding noise.

19. An image generation apparatus, comprising: The text acquisition module is used to acquire target text; The image generation module is used to generate a multi-channel target image based on the target text using a preset target network model; The multi-channel target image has a color channel and an additional channel, the additional channel corresponding to additional information other than color information; the target network model includes a target diffusion network, the input channels of the target diffusion network include a first type of input channel corresponding to the color channel and a second type of input channel corresponding to the additional channel.

20. An electronic device, comprising: A storage device on which computer programs are stored; A processing device for executing the computer program in the storage device to implement the steps of the image generation method according to any one of claims 1-18.

21. A computer-readable storage medium storing a computer program for performing the image generation method according to any one of claims 1-18.

Images