Method and device for training neural network and neural network

Abstract

The invention discloses a neural network training method and device and a neural network. The neural network includes a text encoder, a time encoder, first to third Unet encoders, a Unet decoder, and first and second adders. The method includes: obtaining an input image by adding noise to an original image; inputting the input image into a first Unet encoder, and inputting a line draft image corresponding to the original image and the input image into a second Unet encoder after being added by a first adder; adding the input image and the draft image through a second adder, and then inputting the added image and draft image into a third Unet encoder; and training the neural network by adjusting the parameters of the third Unet encoder and keeping the parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder and the second Unet encoder unchanged, or the parameters of the Unet decoder and the third Unet encoder are adjusted, and the parameters of the text encoder, the time encoder, the first Unet encoder and the second Unet encoder are kept unchanged.

Description

Technical Field

[0001] The present disclosure relates to the field of neural networks, and in particular to style adaptation of line art coloring of images using neural networks. Background Art

[0002] Coloring line drawings is an important step in the creation process of cartoon animation. However, manual coloring is time-consuming and tedious. Therefore, methods for automatically coloring line drawings in a desired style are gaining more and more attention.

[0003] Traditional methods include, for example, directly using an SD (Stable Diffusion) model or a ControlNet model for reasoning, fine-tuning a pre-trained SD model or ControlNet model, or using ControlNet and reasoning based on a reference image (reference-only).

[0004] Directly using the SD (Stable Diffusion) model or ControlNet model for inference cannot generate images with the desired style because the knowledge of the new style is not learned.

[0005] Fine-tuning a pre-trained SD model or ControlNet model may forget the knowledge of previous styles when fine-tuning on new data.

[0006] Reference-only methods, while generalizing well to out-of-distribution styles, may produce low-quality images. Summary of the invention

[0007] A brief overview of the disclosure is provided below in order to provide a basic understanding of certain aspects of the disclosure. It should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0008] According to one aspect of the present disclosure, there is provided a neural network, including: a text encoder, which is input with a prompt to obtain a tensor of prompts; a time encoder, which is input with a time step to obtain a tensor of time steps; a first Unet (U-shaped network) encoder, a second Unet encoder, and a third Unet encoder; a Unet decoder, the output of the first Unet encoder is connected to the first input of the Unet decoder, the output of the second Unet encoder and the output of the third Unet encoder are connected to the second input of the Unet decoder; a first adder and a second adder, the first adder is connected to the input of the second Unet encoder, and the second adder is connected to the input of the third Unet encoder; and a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output of the third Unet encoder and the second input of the Unet decoder. The text encoder and the time encoder are connected to the first Unet encoder, the second Unet encoder, the third Unet encoder, and the Unet decoder. The neural network is trained by adjusting the parameters of the third Unet encoder and keeping the respective parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder and the second Unet encoder unchanged so that the neural network converges, or adjusting the respective parameters of the Unet decoder and the third Unet encoder and keeping the respective parameters of the text encoder, the time encoder, the first Unet encoder and the second Unet encoder unchanged so that the neural network converges.

[0009] Preferably, the neural network further comprises a third zero convolution module connected to the second adder.

[0010] Preferably, the input image is obtained by adding noise to the original image.

[0011] Preferably, during training, the input image is input into the first Unet encoder, the line draft image corresponding to the original image and the input image are added through the first adder and then input into the second Unet encoder, the line draft image is zero-convolved through the first zero convolution module, and the first style reference image is convolved through the third zero convolution module, and the zero-convolved line draft image, the zero-convolved first style reference image and the input image are input into the second adder and added, and then input into the third Unet encoder.

[0012] Preferably, during inference, the input image is input into the first Unet encoder, the line draft image and the input image are added through the first adder and then input into the second Unet encoder, the line draft image is zero-convolved through the first zero convolution module, and the second style reference image is convolved through the third zero convolution module, and the zero-convolved line draft image, the zero-convolved second style reference image and the input image are input into the second adder and added, and then input into the third Unet encoder.

[0013] Preferably, the first style reference image is obtained by performing a Thin-Plate Splines (TPS) transformation on the original image, and the second style reference image is any other image different from the original image.

[0014] Preferably, the neural network further comprises a base transform block, and the base transform block is included in the Unet decoder.

[0015] Preferably, the neural network is a style adaptation neural network.

[0016] Preferably, the style includes one or more of color, brightness, saturation, contrast, texture and lines of the image.

[0017] Preferably, before the input image is input into the first Unet encoder, the first adder and the second adder, the latent variable of the input image is obtained by a pre-trained encoder.

[0018] Preferably, the line drawing image is encoded by an encoder participating in the training of a neural network before being input into the first adder and the first zero convolution module.

[0019] Preferably, before the style reference image is input into the third zero-convolution module, a latent variable of the style reference image is obtained through a pre-trained encoder.

[0020] Preferably, the first Unet encoder, the second Unet encoder and the Unet decoder are pre-trained before training the neural network. Preferably, the network parameters obtained by pre-training the second Unet encoder can be copied as the initial parameters of the third Unet encoder.

[0021] According to another aspect of the present disclosure, there is provided a method for training the above-mentioned neural network, comprising: obtaining an input image by adding noise to an original image; inputting the input image into a first Unet encoder, and inputting a line drawing image corresponding to the original image and the input image into a second Unet encoder after adding them through a first adder; inputting the input image and the line drawing image into a third Unet encoder after adding them through a second adder; and training the neural network by: adjusting the parameters of the third Unet encoder, and keeping the respective parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges, or adjusting the respective parameters of the Unet decoder and the third Unet encoder, and keeping the respective parameters of the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges.

[0022] Preferably, the text encoder and the time encoder are connected to the first Unet encoder, the second Unet encoder, the third Unet encoder and the Unet decoder, and the method further comprises: inputting the prompt into the text encoder and the time step into the time encoder to obtain respective tensors of the prompt and the time step; and providing the respective tensors of the prompt and the time step to the first Unet encoder, the second Unet encoder, the third Unet encoder and the Unet decoder respectively.

[0023] Preferably, the neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output of the third Unet encoder and the second input of the Unet decoder. The line draft image is zero-convolved through the first zero convolution module, and the input image and the zero-convolved line draft image are added through the second adder and input into the third Unet encoder.

[0024] Preferably, the neural network further comprises a third zero convolution module, and the third zero convolution module is connected to the second adder. The method further comprises subjecting the style reference image to zero convolution through the third zero convolution module, and inputting the zero-convolved style reference image into the second adder.

[0025] Preferably, during training, the style reference image is obtained by performing a thin plate spline transformation on the original image.

[0026] Preferably, during inference, the style reference image is any other image different from the original image.

[0027] According to yet another aspect of the present disclosure, there is provided an apparatus for training the above-mentioned neural network, comprising: an obtaining unit configured to obtain an input image by adding noise to an original image; a first input unit configured to input the input image into a first Unet encoder, and input a line draft image corresponding to the original image and the input image into a second Unet encoder after adding them through a first adder; a second input unit configured to input the input image and the line draft image into a third Unet encoder after adding them through a second adder; and a training unit configured to train the neural network by adjusting parameters of the third Unet encoder, and keeping respective parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges, or adjusting respective parameters of the Unet decoder and the third Unet encoder, and keeping respective parameters of the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges.

[0028] According to other aspects of the present disclosure, corresponding computer program codes, computer-readable storage media and computer program products are also provided.

[0029] The method and apparatus for training a neural network of the present disclosure enable the generation of high-quality color images with a desired style.

[0030] These and other advantages of the present disclosure will become more apparent through the following detailed description of preferred embodiments of the present disclosure in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to further illustrate the above and other advantages and features of the present disclosure, the specific embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings. The drawings together with the following detailed description are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only describe typical examples of the present disclosure and should not be regarded as limiting the scope of the present disclosure. In the drawings:

[0032] Figure 1 schematically illustrates a neural network according to an embodiment of the present disclosure;

[0033] Figure 2 The forward process and the reverse process of the Denoising Diffusion Probability Model (DDPM) are schematically shown;

[0034] Figure 3The Latent Diffusion Model (LDM) is schematically shown;

[0035] Figure 4 A flowchart of a method for training a neural network according to an embodiment of the present disclosure is shown;

[0036] Figure 5 Schematically illustrates the output effect obtained by training a neural network according to an embodiment of the present disclosure;

[0037] Figure 6 is a block diagram of an apparatus for training a neural network according to an embodiment of the present disclosure;

[0038] Figure 7 is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or devices according to embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.

[0040] It is also necessary to explain here that, in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not very relevant to the present disclosure are omitted.

[0041] As mentioned above, existing methods for coloring line drawing images either generate low-quality images or fail to generate images with the desired style.

[0042] In order to overcome the shortcomings of the prior art, the present disclosure proposes a style adaptation neural network that can learn new styles of characters, objects, backgrounds, etc., and generate color images from line drawings.

[0043] It should be understood that style includes one or more of color, brightness, saturation, contrast, texture, and lines of an image.

[0044] The style adaptation neural network according to the present disclosure is based on a pre-trained model, such as ControlNet v1.1 Anime Lineart, to utilize the image knowledge of the pre-trained model; use different neural network branches to control the line draft and style respectively; and add new neural network branches to learn and control new image styles, while freezing the weights of the pre-trained model to avoid catastrophic forgetting problems.

[0045] Figure 1 Schematically shows a style adaptation neural network 100 according to an embodiment of the present disclosure. Figure 1 As shown, the style adaptation neural network 100 includes Unet encoders 11, 12, 13, a text encoder 10, a temporal encoder 20, a Unet decoder 30, and adders 40 and 50. The output of the Unet encoder 11 is connected to the first input of the Unet decoder 30. The output of the adder 40 is connected to the input of the Unet encoder 12, and the output of the adder 50 is connected to the input of the Unet encoder 13. The outputs of the Unet encoders 12 and 13 are connected to the second input of the Unet decoder 30. The output of the Unet decoder 30 outputs the inference result of the style adaptation neural network 100.

[0046] It should be understood that the text encoder 10 and the temporal encoder 20 are connected to each of the Unet encoders 11 , 12 , 13 and the Unet decoder 30 .

[0047] As known, the cue Ct is input to the text encoder 10 to obtain a tensor of cues, and the time step t is input to the temporal encoder 20 to obtain a tensor of time steps. The respective tensors of cue Ct and time step t are provided to each of the Unet encoders 11, 12, 13 and the Unet decoder 30, respectively.

[0048] It should also be understood that although Figure 1 , a text encoder 10 and a time encoder 20 are shown, but the present disclosure is not limited thereto, and a plurality of text encoders 10 and a plurality of time encoders 20 may also be provided to be used for each of the Unet encoders 11, 12, 13 and the Unet decoder 30, respectively.

[0049] According to a preferred embodiment, the style adaptation neural network 100 further comprises zero convolution modules 21, 23. The output of the zero convolution module 21 is connected to the input of the adder 50, and the zero convolution module 23 is connected between the output of the Unet encoder 13 and the second input of the Unet decoder.

[0050] According to another preferred embodiment, the style adaptation neural network 100 further includes a zero convolution module 22. The output of the zero convolution module 22 is connected to the input of the adder 50.

[0051] During the training of the style adaptation neural network 100, the original image is firstly noised to obtain a noisy input image 1, and then the latent variable z of the noisy input image 1 is obtained. t Hidden variable z t is input to the Unet encoder 11, adder 40 and adder 50. The line drawing image C corresponding to the original image f is input to adder 40, and is input to adder 50 after zero convolution by zero convolution module 21. Hidden variable z t With line drawing image C f After being added by the adder 40, the result is input into the Unet encoder 12. t And the line drawing image C after zero convolution f The outputs of the Unet encoder 11 and 12 are respectively provided to the first input and the second input of the Unet decoder 30, and the output of the Unet encoder 13 is provided to the second input of the Unet decoder 30 after being zero-convolved by the zero-convolution module 23.

[0052] During training, only the parameters of the Unet encoder 13 may be adjusted while keeping the parameters of the Unet decoder 30, text encoder 10, temporal encoder 20, and Unet encoders 11 and 12 unchanged, so that the style adaptation neural network 100 converges.

[0053] According to a preferred embodiment, during training, the respective parameters of the Unet encoder 13 and the Unet decoder 30 can be adjusted while keeping the respective parameters of the text encoder 10, the temporal encoder 20, and the Unet encoders 11 and 12 unchanged, so that the style adaptation neural network 100 converges.

[0054] It should be understood that parameters include weights and biases.

[0055] According to a preferred embodiment, during training, the style reference image 2' can also be provided to the adder 50 after being zero-convolved by the zero-convolution module 22. The style reference image 2' is obtained after the style reference image 2 is TPS-converted. In this case, the adder 50 adds the latent variable z t , Line drawing image C after zero convolution f It is added to the style reference image 2 ′ after zero convolution, and the added result is provided to the Unet encoder 13 .

[0056] It should be noted that during training, the style reference image 2 corresponds to the original image of the noisy input image 1.

[0057] It should be understood that the zero convolution modules 21, 22 and 23 mean that their weights are 0 only during the first iteration of training of the style adaptation neural network 100, and are not zero during subsequent iterations of training and during inference.

[0058] It should also be understood that the line drawing image C f Before being input to the adder 40 and the zero convolution module 21, the style reference image 2' is encoded by an encoder (not shown) participating in the neural network training, and before being input to the zero convolution module 22, the style reference image 2' is obtained by a pre-trained encoder (not shown) to obtain its latent variable C s The above encoding of the line drawing image and the style reference image may use any suitable existing technology, and will not be described in detail here.

[0059] It should also be understood that the original image can be noised using the existing DDPM to obtain the noisy input image 1, and the original image can also be converted into a latent variable z by using the pre-trained autoencoder in the LDM t And then the latent variable is added with noise by DDPM. In order to facilitate understanding of the present disclosure, the process of adding noise to the original image and obtaining the latent variable of the image after adding noise is briefly introduced below.

[0060] Figure 2 The forward process and reverse process of DDPM are schematically shown. Figure 2 As shown, in the forward process from right to left, from X0 to X T Noise is continuously added to the image for diffusion, thus obtaining a noisy image. In the reverse process from left to right, from X T To X0, the noise is gradually removed from the noisy image to obtain a noise-free image. The forward process and the reverse process can occur in both the pixel space and the latent space.

[0061] According to an embodiment of the present disclosure, a DDPM-based LDM is used to perform forward diffusion and reverse denoising processes in a latent space.

[0062] Figure 3 Schematically shows a block diagram of LDM 300. Figure 3 As shown, LDM 300 includes latent space 301, condition module 302, diffusion module 303, denoising module 304 and pixel space 305.

[0063] First, the original image x0 is mapped to the latent space 301 through the pre-trained encoder to obtain the latent variable z0, and then the forward diffusion process is performed through the diffusion module 303 to obtain zT , z T Add or concatenate to obtain a new tensor, and provide the new tensor to the denoising module 304 for the reverse denoising process. In the reverse process, after finally obtaining the noise-free latent variable z0, it is mapped back to the pixel space 305 through the pre-trained decoder to obtain the noise-free image x0. The condition module 302 is used to provide the denoising module 304 with conditions of the type of image, text, and semantic map.

[0064] The following two diffusion processes may be implemented in LDM 300 .

[0065] In the first diffusion process, a noise-free latent variable z0 is first given, and then the diffusion process is performed step by step. After performing t steps of diffusion, the noisy latent variable z t :

[0066] z t = z t-1 +∈=z0+∈0+∈1+…+∈ (1)

[0067] Among them, ∈0 is the noise added in step 1, ∈1 is the noise added in step 2, and ∈ is the noise added in step t. However, this is inefficient because it requires multiple sampling calculations to obtain z t .

[0068] Advantageously, in the second diffusion process, we can use the derivation of DDPM reparameterization to obtain z directly from z0 by sampling the noise only once. t :

[0069]

[0070] where ∈ is the noise sampled from a Gaussian distribution, is a preset constant and becomes smaller and smaller as t increases. As t increases, z becomes smaller and smaller. t The content of the original image z0 contained in becomes less and less, while the noise ∈ becomes more and more.

[0071] The above briefly introduces the input latent variable z of the style adaptation neural network 100 t The loss function L of the style adaptation neural network 100 is introduced below.

[0072] Without providing a style reference image 2', the loss function of the style adaptation neural network 100 is as follows:

[0073]

[0074] Among them, z0 represents the hidden variable corresponding to the image without noise, t represents the time step of random sampling, and c t Indicates text condition (text prompt condition), c f Indicates the line draft condition, represents the noise ∈ sampled from a Gaussian distribution with mean 0 and variance 1.

[0075] Adding noise ∈ to z0 gives z t .Right now: in is a preset constant. Noise ∈ is taken as a label (ground truth) as the target that the style adaptation neural network 100 needs to predict. The network parameter that needs to be optimized in the style adaptation neural network 100 is θ. As mentioned above, the network parameter θ that needs to be optimized involves the Unet encoder 13, and preferably also involves the Unet decoder 30.

[0076] During training, z t ,t,c t ,c f Input style adaptation neural network 100 to predict z t The noise contained in θ (z t ,t,c t ,c f ). Next, calculate ∈ and ∈ θ (z t ,t,c t ,c f )’s L2 loss (square loss function) is used as the loss function L of the style adaptation neural network 100.

[0077] When a style reference image 2' is provided, the loss function L' of the style adaptation neural network 100 is as follows:

[0078]

[0079] Among them, c s Indicates style conditions.

[0080] It should be noted that during training, c s The original image x0 is transformed by TPS to obtain the deformed style reference image x′0=TPS(x0), and then c is obtained by the encoder. s During inference, c s It can be obtained by inputting any image that is unrelated or related to the original image into the encoder.

[0081] It should be understood that optimizing the network parameters θ includes calculating the gradient of the network parameters according to the loss function L or L′ where w∈{d,e,z1,z2,z3}, and then update the parameter θ according to the stochastic gradient descent algorithm W : Where μ is the preset learning rate.

[0082] By optimizing the network parameters, the style adaptation neural network 100 is trained until convergence (ie, the loss function no longer decreases) or a preset number of training rounds is reached.

[0083] It should be noted that before training the style adaptation neural network 100, the Unet encoders 11, 12 and the Unet decoder 30 are pre-trained.

[0084] During the inference of the style adaptation neural network 100, the prompt "Gauguin style", the line drawing and latent variables corresponding to the original image, and the style reference image are input into the trained style adaptation neural network 100, and a color image of the line drawing that matches the prompt and the style reference image can be obtained at the output of the Unet decoder 30.

[0085] It should be understood that the content of the prompt is not limited to the style of the image, but may include any text that describes the content to be generated.

[0086] It should also be understood that line drawings extracted from the original image may be provided, or line drawings directly obtained may be provided.

[0087] It should also be understood that during inference, the style reference image can be any image that is independent or related to the original image.

[0088] It should also be understood that Figure 2 The style adaptation neural network 100 also includes a basic transformation block (BasicTransformerBlock) (not shown), and the basic transformation block can be considered to be included in the Unet decoder 30.

[0089] Although the purpose of generating a certain painter's style can be achieved by controlling only through prompts, a painter may have multiple sub-styles. However, sub-styles cannot be accurately described by words. Therefore, what is generated through prompts is just a random one of the multiple sub-styles. By additionally inputting a style reference image, an image with a desired sub-style can be better generated.

[0090] Combine the following Figure 1 The style adaptation neural network 100 is shown to describe Figure 4 A method 400 for training a neural network according to an embodiment of the present invention is shown.

[0091] First, in step 401, an input image is obtained by adding different noises to the original image. Specifically, in this embodiment, using Figure 3 The LDM 300 shown converts the original image into a noisy input image 1, and then obtains the latent variable z of the noisy input image 1. t .

[0092] Next, in step 402, the input image is input into the first U-shaped network encoder, and the line drawing image corresponding to the original image and the input image are added by the first adder and then input into the second U-shaped network encoder. Specifically, in this embodiment, the latent variable z of the noisy input image 1 is t The Unet encoder 11 is input, and the line drawing image C encoded by the encoder (not shown) participating in the training is f and the latent variable z t After being added by the adder 40 , the signals are input to the Unet encoder 12 .

[0093] Next, in step 403, the input image and the line drawing image are input to the second adder. Specifically, in this embodiment, the latent variable z t and the encoded line drawing image C f Input adder 50.

[0094] Advantageously, the encoded line drawing image C f The zero convolution module 21 may perform zero convolution and then provide the result to the adder 50 .

[0095] Next, in step 404, the second adder inputs the result of adding the input image and the line drawing image into the third U-shaped network encoder. Specifically, in this embodiment, the adder 50 adds the latent variable z t and the encoded line drawing image C f The sum is input to the Unet encoder 13.

[0096] Finally, in step 405, the neural network is trained. Specifically, in this embodiment, when training the style adaptation neural network 100, only the parameters of the Unet encoder 13 may be adjusted, while the parameters of the Unet decoder 30, the text encoder 10, the time encoder 20, and the Unet encoders 11 and 12 are kept unchanged, so that the style adaptation neural network 100 converges.

[0097] Preferably, when training the style adaptation neural network 100, the respective parameters of the Unet encoder 13 and the Unet decoder 30 can be adjusted, while the respective parameters of the text encoder 10, the time encoder 20, and the Unet encoders 11 and 12 are kept unchanged, so that the style adaptation neural network 100 converges.

[0098] In this embodiment, the loss function L of the style adaptation neural network 100 is as shown in the above formula (3).

[0099] In a preferred embodiment, the style reference image may be input into the second adder in step 403'. Specifically, in this embodiment, the latent variable C of the style reference image 2' obtained by the pre-trained encoder (not shown) s is also input to adder 50, and adder 50 adds the hidden variable z t , Encoded line drawing image C f and the latent variable C of the style reference image 2' s The sum is input to the Unet encoder 13.

[0100] In this preferred embodiment, the loss function L′ of the style adaptation neural network 100 is as shown in the above formula (4).

[0101] Advantageously, the latent variable C of the style reference image 2' s It may be provided to the adder 50 after being zero-convolved by the zero-convolution module 22 .

[0102] It should be noted that the style reference Figure 2 ' is obtained by performing TPS conversion on the style reference image 2.

[0103] It should also be noted that during training, the style reference image 2 corresponds to the noisy input image 1 and the line drawing C f During inference, the style reference image 2 can be the original image of the noisy input image 1 and the line drawing C f Any image that is related or unrelated to the original image.

[0104] Advantageously, during training, the output of the Unet encoder 13 may be input to a second input of the Unet decoder 30 after being zero-convolved by the zero-convolution module 23 .

[0105] pass Figure 4 The illustrated method 400 for training a neural network enables the generation of high quality color images having a desired style.

[0106] Figure 5 Schematically shows the use of Figure 4 Method 400 training Figure 1 After the style adaptation neural network 100 is shown, the output image quality obtained by using the trained style adaptation neural network 100.

[0107] like Figure 5As shown, for example, the prompt input to the text encoder 10 (not shown) is "Gauguin style", and the style reference image 2' corresponds to a sub-style of Gauguin. t and line drawing image C f After having a reference image 2' with a higher style, the style of the image output by the Unet decoder 30 matches well with the style reference image 2'.

[0108] The methods discussed above may be implemented entirely by a computer executable program, or may be implemented partially or entirely using hardware and / or firmware. When implemented in hardware and / or firmware, or when a computer executable program is loaded into a hardware device that can run the program, the apparatus for training a neural network described below is implemented. Below, an overview of these apparatuses is given without repeating some of the details discussed above, but it should be noted that although these apparatuses can perform the methods described above, the methods do not necessarily employ those components of the described apparatuses or are not necessarily performed by those components.

[0109] Figure 6 An apparatus 600 for training a neural network according to an embodiment of the present disclosure is shown. The apparatus 600 includes an obtaining unit 601, a first input unit 602, a second input unit 603, and a training unit 604. The obtaining unit 601 is used to obtain an input image by adding noise to an original image. The first input unit 602 is used to input the input image into a first Unet encoder, and input the line drawing image corresponding to the original image and the input image into a second Unet encoder after adding them through a first adder. The second input unit 603 is used to input the input image and the line drawing image into a third Unet encoder after adding them through a second adder. The training unit 604 is used to train the neural network by adjusting the parameters of the third Unet encoder, and keeping the parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges; or adjusting the parameters of the Unet decoder and the third Unet encoder, and keeping the parameters of the text encoder, the time encoder, the first Unet encoder, and the second Unet encoder unchanged, so that the neural network converges.

[0110] Figure 6 The apparatus 600 for training a neural network shown corresponds to Figure 4 Therefore, the relevant details of each unit in the apparatus 600 for training a neural network have been described in detail. Figure 4 The method 400 for training a neural network is described in detail and will not be repeated here.

[0111] Each component module or unit in the above device can be configured by software, firmware, hardware or a combination thereof. The specific means or methods that can be used for configuration are well known to those skilled in the art and will not be described in detail here. In the case of software or firmware implementation, the data is transmitted from a storage medium or a network to a computer (e.g., a computer) with a dedicated hardware structure. Figure 7 The general-purpose computer 700 shown in the figure is installed with the programs constituting the software, and when the various programs are installed, the computer can execute various functions, etc.

[0112] Figure 7 is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or devices according to embodiments of the present invention may be implemented. Figure 7 As shown, a central processing unit (CPU) 701 performs various processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, data required when the CPU 701 performs various processes and the like is also stored as needed. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0113] The following components are connected to the input / output interface 705: an input device 706 (including a keyboard, a mouse, etc.), an output device 707 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage device 708 (including a hard disk, etc.), a communication device 709 (including a network interface card such as a LAN card, a modem, etc.). The communication device 709 performs communication processing via a network such as the Internet. A drive 710 may also be connected to the input / output interface 705 as needed. A removable medium 711 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 710 as needed, so that a computer program read therefrom is installed in the storage device 708 as needed.

[0114] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 711 .

[0115] It should be understood by those skilled in the art that such storage media is not limited to Figure 7The removable medium 711 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 711 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 702, a hard disk included in the storage device 708, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0116] The present disclosure also provides corresponding computer program codes and a computer program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the method 400 according to the implementation of the present disclosure can be executed.

[0117] Accordingly, the storage medium configured to carry the program product storing the machine-readable instruction code is also included in the disclosure of the present disclosure, including but not limited to a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0118] Through the above description, the embodiments of the present disclosure provide the following technical solutions, but are not limited thereto.

[0119] Note 1. A method for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a U-shaped network decoder, a first adder and a second adder, the output of the first U-shaped network encoder is connected to the first input of the U-shaped network decoder, the first adder is connected to the input of the second U-shaped network encoder, the second adder is connected to the input of the third U-shaped network encoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to the second input of the U-shaped network decoder, the method comprising:

[0120] The input image is obtained by adding noise to the original image;

[0121] Input the input image into the first U-shaped network encoder, and input the line drawing image corresponding to the original image and the input image into the second U-shaped network encoder after adding them through the first adder;

[0122] adding the input image and the line drawing image through the second adder and inputting them into the third U-shaped network encoder; and

[0123] The neural network is trained as follows:

[0124] Adjust the parameters of the third U-shaped network encoder and keep the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or adjust the parameters of the U-shaped network decoder and the third U-shaped network encoder and keep the parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

[0125] Note 2. The method according to Note 1, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, and the method further comprises:

[0126] Inputting a prompt into the text encoder and a time step into the temporal encoder to obtain respective tensors of the prompt and the time step; and

[0127] The respective tensors of the hint and the time step are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, respectively.

[0128] Note 3. The method according to Note 2, wherein the neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output of the third U-shaped network encoder and the second input of the U-shaped network decoder, and

[0129] The line drawing image is zero-convolved through the first zero-convolution module, and the input image and the line drawing image that has undergone zero-convolution are added through the second adder and then input into the third U-shaped network encoder.

[0130] Note 4. The method according to Note 3, wherein the neural network further comprises a third zero convolution module, wherein the third zero convolution module is connected to the second adder,

[0131] The method also includes performing zero convolution on the style reference image through the third zero convolution module, and inputting the zero-convolved style reference image into the second adder.

[0132] Note 5. The method according to Note 4, wherein, during training, the style reference image is obtained by performing a thin plate spline method transformation on the original image.

[0133] Note 6. The method according to Note 4, wherein, during inference, the style reference image is any other image different from the original image.

[0134] Note 7. A method according to any one of Notes 1 to 6, wherein, before the input image is input into the first U-shaped network encoder, the first adder and the second adder, the latent variables of the input image are obtained by a pre-trained encoder.

[0135] Note 8. The method according to Note 4, wherein the line drawing image is encoded by an encoder participating in the training of the neural network before being input into the first adder and the first zero convolution module.

[0136] Note 9. The method according to Note 5 or 6, wherein, before the style reference image is input into the third zero convolution module, the latent variables of the style reference image are obtained by a pre-trained encoder.

[0137] Note 10. A method according to any one of Notes 1 to 6, wherein the neural network further comprises a basic transform block, and the basic transform block is included in the U-shaped network decoder.

[0138] Note 11. A method according to any one of Notes 1 to 6, wherein the neural network is a style adaptation neural network, and wherein the style includes one or more of color, brightness, saturation, contrast, texture and lines of the image.

[0139] Note 12. According to the method described in any one of Notes 1 to 6, the first U-shaped network encoder, the second U-shaped network encoder and the U-shaped network decoder are pre-trained before training the neural network. The network parameters obtained by pre-training the second Unet encoder can also be copied as the initial parameters of the third Unet encoder.

[0140] Note 13. A device for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a U-shaped network decoder, a first adder and a second adder, the output of the first U-shaped network encoder is connected to the first input of the U-shaped network decoder, the first adder is connected to the input of the second U-shaped network encoder, the second adder is connected to the input of the third U-shaped network encoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to the second input of the U-shaped network decoder, the device comprising:

[0141] an obtaining unit configured to obtain an input image by adding noise to an original image;

[0142] a first input unit configured to input the input image into the first U-shaped network encoder, and input the line drawing image corresponding to the original image and the input image into the second U-shaped network encoder after adding them through the first adder;

[0143] a second input unit configured to input the input image and the line drawing image into the third U-shaped network encoder after adding them through the second adder; and

[0144] A training unit is configured to train the neural network by:

[0145] Adjust the parameters of the third U-shaped network encoder and keep the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or adjust the parameters of the U-shaped network decoder and the third U-shaped network encoder and keep the parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

[0146] Note 14. A computer-readable storage medium having a program stored thereon, the program, when executed by a processor, causing a computer to implement the method according to any one of Notes 1 to 12.

[0147] Note 15. A neural network comprising:

[0148] a text encoder, the text encoder being input with a prompt to obtain a tensor of the prompt;

[0149] a time encoder that is input with a time step to obtain a tensor for the time step;

[0150] A first U-shaped network encoder, a second U-shaped network encoder, and a third U-shaped network encoder;

[0151] A U-shaped network decoder, wherein the output of the first U-shaped network encoder is connected to a first input of the U-shaped network decoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to a second input of the U-shaped network decoder;

[0152] a first adder and a second adder, the first adder being connected to an input of the second U-shaped network encoder, and the second adder being connected to an input of the third U-shaped network encoder; and

[0153] a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third U-shaped network encoder and a second input of the U-shaped network decoder,

[0154] wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, and

[0155] The neural network is trained in the following way:

[0156] Adjust the parameters of the third U-shaped network encoder, and keep the respective parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or adjust the respective parameters of the U-shaped network decoder and the third U-shaped network encoder, and keep the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

[0157] Note 16. The neural network according to Note 15 further includes a third zero convolution module, wherein the third zero convolution module is connected to the second adder.

[0158] Note 17. The neural network according to Note 16, wherein the input image is obtained by adding noise to the original image,

[0159] Among them, during training:

[0160] Inputting the input image into the first U-shaped network encoder,

[0161] The line drawing image corresponding to the original image and the input image are added by the first adder and then input into the second U-shaped network encoder,

[0162] performing zero convolution on the line drawing image through the first zero convolution module, and performing convolution on the first style reference image through the third zero convolution module, and

[0163] The line drawing image after zero convolution, the first style reference image after zero convolution and the input image are input into the second adder, and then input into the third U-shaped network encoder, and

[0164] Among them, during inference:

[0165] Inputting the input image into the first U-shaped network encoder,

[0166] The line drawing image and the input image are added by the first adder and then input into the second U-shaped network encoder,

[0167] performing zero convolution on the line drawing image through the first zero convolution module, and performing convolution on the second style reference image through the third zero convolution module, and

[0168] The line drawing image that has undergone zero convolution, the second style reference image that has undergone zero convolution, and the input image are input into the second adder, added, and then input into the third U-shaped network encoder.

[0169] Note 18. A neural network according to Note 17, wherein the first style reference image is obtained by performing a thin plate spline method conversion on the original image, and wherein the second style reference image is any other image different from the original image.

[0170] Note 19. The neural network according to any one of Notes 15 to 18 further includes a basic transform block, which is included in the U-shaped network decoder.

[0171] Note 20. A neural network according to any one of Notes 15 to 18, wherein the neural network is a style adaptation neural network.

[0172] Finally, it should be noted that the terms "comprises", "includes" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In addition, in the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, article or device that includes the elements.

[0173] Although the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only configured to illustrate the present disclosure and do not constitute a limitation on the present disclosure. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the attached claims and their equivalents.

Claims

1. A method for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a U-shaped network decoder, a first adder and a second adder, the output of the first U-shaped network encoder is connected to the first input of the U-shaped network decoder, the first adder is connected to the input of the second U-shaped network encoder, the second adder is connected to the input of the third U-shaped network encoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to the second input of the U-shaped network decoder, the method comprising: The input image is obtained by adding noise to the original image; Input the input image into the first U-shaped network encoder, and input the line drawing image corresponding to the original image and the input image into the second U-shaped network encoder after adding them through the first adder; adding the input image and the line drawing image through the second adder and inputting them into the third U-shaped network encoder; and The neural network is trained as follows: adjusting the parameters of the third U-shaped network encoder and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or The respective parameters of the U-shaped network decoder and the third U-shaped network encoder are adjusted, and the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder are kept unchanged, so that the neural network converges.

2. The method according to claim 1, wherein: The text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, and the method further comprises: Inputting a prompt into the text encoder and a time step into the temporal encoder to obtain respective tensors of the prompt and the time step; and The respective tensors of the hint and the time step are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, respectively.

3. The method according to claim 2, wherein: The neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third U-shaped network encoder and a second input of the U-shaped network decoder, and The line drawing image is zero-convolved through the first zero-convolution module, and the input image and the line drawing image that has undergone zero-convolution are added through the second adder and then input into the third U-shaped network encoder.

4. The method according to claim 3, wherein: The neural network further comprises a third zero convolution module, the third zero convolution module being connected to the second adder, The method also includes performing zero convolution on the style reference image through the third zero convolution module, and inputting the zero-convolved style reference image into the second adder.

5. The method according to claim 4, wherein: During training, the style reference image is obtained by performing a thin plate spline transformation on the original image.

6. The method according to claim 4, wherein: During inference, the style reference image is any other image different from the original image.

7. The method according to any one of claims 1 to 6, wherein: The first U-shaped network encoder, the second U-shaped network encoder and the U-shaped network decoder are pre-trained before training the neural network.

8. A device for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a U-shaped network decoder, a first adder and a second adder, the output of the first U-shaped network encoder is connected to the first input of the U-shaped network decoder, the first adder is connected to the input of the second U-shaped network encoder, the second adder is connected to the input of the third U-shaped network encoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to the second input of the U-shaped network decoder, the device comprising: an obtaining unit configured to obtain an input image by adding noise to an original image; a first input unit configured to input the input image into the first U-shaped network encoder, and input the line drawing image corresponding to the original image and the input image into the second U-shaped network encoder after adding them through the first adder; A second input unit, configured to input the input image and the line drawing image into the third U-shaped network encoder after adding them through the second adder; and A training unit is configured to train the neural network by: adjusting the parameters of the third U-shaped network encoder and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or The respective parameters of the U-shaped network decoder and the third U-shaped network encoder are adjusted, and the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder are kept unchanged, so that the neural network converges.

9. A neural network comprising: a text encoder, the text encoder being input with a prompt to obtain a tensor of the prompt; a time encoder that is input with a time step to obtain a tensor for the time step; A first U-shaped network encoder, a second U-shaped network encoder, and a third U-shaped network encoder; A U-shaped network decoder, wherein the output of the first U-shaped network encoder is connected to a first input of the U-shaped network decoder, and the output of the second U-shaped network encoder and the output of the third U-shaped network encoder are connected to a second input of the U-shaped network decoder; a first adder and a second adder, the first adder being connected to an input of the second U-shaped network encoder, and the second adder being connected to an input of the third U-shaped network encoder; and a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third U-shaped network encoder and a second input of the U-shaped network decoder, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder and the U-shaped network decoder, and The neural network is trained in the following way: adjusting the parameters of the third U-shaped network encoder and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged so that the neural network converges, or The respective parameters of the U-shaped network decoder and the third U-shaped network encoder are adjusted, and the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder are kept unchanged, so that the neural network converges.

10. The neural network of claim 9, further comprising a third zero convolution module connected to the second adder.

Images