Method and apparatus for training neural network, and neural network

Abstract

To provide a method and apparatus for training a neural network for painting a line-drawing image with colors in accordance with the style, and a neural network.SOLUTION: A method for training a neural network including a text encode, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder includes: obtaining an input image by adding noise to an original image; inputting the input image to the first U-Net encoder; inputting a line-drawing image corresponding to the original image and the input image, after addition of the first adder, to the second U-Net encoder; inputting the input image and the line-drawing mage to the second adder; inputting a style reference image to the second adder; and inputting the results of the addition to the third U-Net encoder to train the neural network.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to the field of neural networks, and more particularly to style-adaptive coloring of line-art images using neural networks. [Background technology]

[0002] Coloring line-drawing images of animation is an important part of the production process of animation movies. However, coloring by hand is time-consuming and tedious. Therefore, methods for automatically coloring line-drawing images in a desired style have attracted increasing attention.

[0003] Conventional methods include, for example, directly using a Stable Diffusion (SD) model or a ControlNet model for inference, fine-tuning a pre-trained SD model or a ControlNet model, or using ControlNet to perform inference based on reference images (reference-only).

[0004] Inference using the Stable Diffusion (SD) model or the ControlNet model directly cannot generate images of the desired style because they do not have knowledge of the new style.

[0005] Fine-tuning a pre-trained SD model or ControlNet model may forget previous style knowledge when fine-tuning on new data.

[0006] Reference-only methods generalize well to out-of-distribution styles, but can produce poor quality images. Summary of the Invention [Problem to be solved by the invention]

[0007] The following presents a simplified summary of the disclosure in order to provide a basic understanding of aspects of the disclosure. It is not intended to be an exhaustive overview of the disclosure, nor is it intended to intentionally identify key or critical portions of the disclosure or to intentionally limit the scope of the disclosure. However, it is merely intended to introduce concepts in a simplified form as a prelude to the more detailed description that is presented later.

[0008] The present disclosure provides a method, an apparatus and a neural network for training a neural network. [Means for solving the problem]

[0009] In one aspect of the present disclosure, a neural network includes a text encoder to which a prompt is input to obtain a tensor of the prompt, a time encoder to which a time step is input to obtain a tensor of a time step, a first U-Net encoder, a second U-Net encoder, and a third U-Net encoder, a U-Net decoder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder, a first adder connected to an input of the second U-Net encoder, and a second adder connected to an input of the third U-Net encoder, a first zero convolution module connected to the second adder, and a second adder connected between an output of the third U-Net encoder and the second input of the U-Net decoder. and a second zero convolution module connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder, and the U-Net decoder, wherein when the neural network is trained, a parameter of the third U-Net encoder is adjusted so that the neural network converges, and parameters of the U-Net decoder, the text encoder, the temporal encoder, the first U-Net encoder, and the second U-Net encoder are kept unchanged, or parameters of the U-Net decoder and the third U-Net encoder are adjusted so that the neural network converges, and parameters of the text encoder, the temporal encoder, the first U-Net encoder, and the second U-Net encoder are kept unchanged.

[0010] Preferably, the neural network further includes a third zero-convolution module, said third zero-convolution module being connected to said second adder.

[0011] Preferably, noise is added to the original image to obtain the input image.

[0012] Preferably, during training, the input image is input to the first U-Net encoder, a line drawing image corresponding to the original image and the input image are added by the first adder, and then input to the second U-Net encoder, the line drawing image is zero-convolved by the first zero-convolution module, the first style reference image is convolved by the third zero-convolution module, and the zero-convolved line drawing image, the zero-convolved first style reference image, and the input image are input to the second adder and added, and then input to the third U-Net encoder.

[0013] Preferably, during inference, the input image is input to the first U-Net encoder, the line drawing image and the input image are added by the first adder, and then input to the second U-Net encoder, zero convolution is performed on the line drawing image by the first zero convolution module, convolution is performed on the second style reference image by the third zero convolution module, and the zero-convolved line drawing image, the zero-convolved second style reference image, and the input image are input to the second adder and added, and then input to the third U-Net encoder.

[0014] Preferably, the first style-referenced image is obtained by performing a Thin-Plate Splines (TPS) transformation on the original image, and the second style-referenced image is any other image different from the original image.

[0015] Preferably, the neural network further includes a basic transformation block included in the U-Net decoder.

[0016] Preferably, the neural network is a style adaptive neural network.

[0017] Preferably, said style includes one or more of the colour, brightness, saturation, contrast, texture and line of an image.

[0018] Preferably, before the input image is input to the first U-Net encoder, the first adder and the second adder, latent variables of the input image are obtained by a pre-trained encoder.

[0019] Preferably, the line drawing image is encoded by an encoder which participates in training the neural network before being input to the first adder and the first zero convolution module.

[0020] Preferably, before the style reference image is input to the third zero convolution module, latent variables of the style reference image are obtained by a pre-trained encoder.

[0021] Preferably, the first U-Net encoder, the second U-Net encoder and the U-Net decoder are pre-trained before training the neural network. Preferably, the network parameters obtained by pre-training the second U-Net encoder may be replicated as initial parameters of a third U-Net encoder.

[0022] In another aspect of the present disclosure, there is provided a method for training a neural network, the neural network including a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder, the method comprising the steps of: inputting the input image to the first U-Net encoder, and obtaining an input image by adding a line drawing image corresponding to the original image and the input image by the first adder, and obtaining an input image by adding the line drawing image corresponding to the original image and the input image by the first adder. a step of inputting the input image and the line drawing image to a U-Net encoder; a step of adding the input image and the line drawing image by the second adder and then inputting the added image to the third U-Net encoder; and a step of training the neural network, wherein, when training the neural network, parameters of the third U-Net encoder are adjusted so that the neural network converges, and parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder are maintained as they are, or parameters of the U-Net decoder and the third U-Net encoder are adjusted so that the neural network converges, and parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder are maintained as they are.

[0023] Preferably, the text encoder and the temporal encoder are connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder and the U-Net decoder, and the method further includes the steps of inputting a prompt to the text encoder and inputting a time step to the temporal encoder to obtain tensors for the prompt and the time step, and providing the tensors for the prompt and the time step to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder and the U-Net decoder, respectively.

[0024] Preferably, the neural network further includes a first zero-convolution module and a second zero-convolution module, the first zero-convolution module is connected to the second adder, the second zero-convolution module is connected between the output of the third U-Net encoder and a second input of the U-Net decoder, the line drawing image is zero-convolved by the first zero-convolution module, and the input image and the zero-convolved line drawing image are added by the second adder before being input to the third U-Net encoder.

[0025] Preferably, the neural network further includes a third zero-convolution module, the third zero-convolution module being connected to the second adder, and the method further includes the step of zero-convolving a style reference image by the third zero-convolution module and inputting the zero-convolved style reference image to the second adder.

[0026] Preferably, during training, the style reference images are obtained by performing a thin-plate spline transformation on the original images.

[0027] Preferably, during inference, the style reference image is any other image different from the original image.

[0028] In yet another aspect of the present disclosure, there is provided an apparatus for training a neural network, the neural network including a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder, the apparatus further includes an acquisition unit that adds noise to an original image to acquire an input image, and an input unit that inputs the input image to the first U-Net encoder, and adds a line drawing image corresponding to the original image and the input image by the first adder, and then acquires the second U-Net decoder. The present invention provides an apparatus including a first input unit that inputs an et encoder to a U-Net encoder, a second input unit that adds the input image and the line drawing image using the second adder and then inputs the added image to the third U-Net encoder, and a training unit that trains the neural network, wherein the training unit, when training the neural network, adjusts parameters of the third U-Net encoder so that the neural network converges, and keeps the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged, or adjusts parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges, and keeps the parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged.

[0029] Other aspects of the present disclosure further provide corresponding computer program code, a computer readable storage medium, and a computer program product.

[0030] The neural network training method and apparatus disclosed herein can generate high quality color images with a desired style.

[0031] The above and other advantages of the present disclosure will become more apparent from the following detailed description of preferred embodiments of the present disclosure with reference to the drawings. [Brief description of the drawings]

[0032] In order to make the above and other advantages and features of the present disclosure comprehensible, the following detailed description of specific embodiments of the present disclosure will be described in detail with reference to the drawings. The drawings and the following detailed description are incorporated in and constitute a part of this specification. Elements having the same functions and structures are indicated by the same reference numerals. It should be noted that these drawings are merely for illustrating typical examples of the present disclosure and are not intended to limit the scope of the present disclosure. [Figure 1] FIG. 1 is a schematic diagram illustrating a neural network according to an embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram showing the forward and backward processes of the Denoising Diffusion Probability Model (DDPM). [Diagram 3] FIG. 1 is a schematic diagram showing a Latent Diffusion Model (LDM). [Figure 4] 1 is a flow chart illustrating a method for training a neural network according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a schematic diagram illustrating the output effect obtained by training a neural network according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a block diagram of a neural network training device according to an embodiment of the present disclosure. [Figure 7] FIG. 1 is a block diagram illustrating an exemplary configuration of a general purpose personal computer in which methods and / or apparatus according to embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0033] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. For convenience of description, the specification does not show all the features of the actual embodiment. In addition, in order to realize the specific goal of the developer in the actual implementation, the specific embodiment may be modified, for example, the embodiment may be modified according to the constraints on the system and business. In addition, although the development work is very complicated and time-consuming, for those skilled in the art of this disclosure, this development work is merely an example work.

[0034] It should be noted that, for clarity of the present disclosure, the drawings show only the device components and / or process steps closely related to the present disclosure, and omit details unrelated to the present disclosure.

[0035] As mentioned above, conventional methods of coloring line art images produce poor quality images or fail to produce images in the desired style.

[0036] To address the problems of the prior art, the present disclosure provides a style-adaptive neural network that can learn new styles, such as people, objects, and backgrounds, and generate color images from line drawings.

[0037] The style includes one or more of the color, brightness, saturation, contrast, texture, and line of the image.

[0038] The style adaptation neural network of the present disclosure is based on a pre-trained model, such as ControlNet v1.1 Anime Lineart, and utilizes the image knowledge of the pre-trained model. Different neural network branches are used to control the line art and style, respectively. New neural network branches are added to learn and control new image styles, and the weights of the pre-trained model are frozen to avoid the problem of catastrophic forgetting.

[0039] FIG. 1 is a schematic diagram illustrating a style adaptive neural network 100 according to an embodiment of the present disclosure. As shown in FIG. 1, the style adaptive neural network 100 includes U-Net encoders 11, 12, and 13, a text encoder 10, a time encoder 20, a U-Net decoder 30, and adders 40 and 50. The output of the U-Net encoder 11 is connected to a first input of the U-Net decoder 30. The output of the adder 40 is connected to an input of the U-Net encoder 12, and the output of the adder 50 is connected to an input of the U-Net encoder 13. The outputs of the U-Net encoders 12 and 13 are connected to a second input of the U-Net decoder 30. The output of the U-Net decoder 30 outputs the inference result of the style adaptive neural network 100.

[0040] In addition, the text encoder 10 and the time encoder 20 are connected to the U-Net encoders 11, 12, and 13 and the U-Net decoder 30, respectively.

[0041] Also, prompt C t is input to the text encoder 10 to obtain a tensor of prompts, and the time step t is input to the time encoder 20 to obtain a tensor of time steps. t The tensors for time step t are provided to U-Net encoders 11, 12, 13 and U-Net decoder 30, respectively.

[0042] Although FIG. 1 shows one text encoder 10 and one temporal encoder 20, the present disclosure is not limited thereto, and multiple text encoders 10 and multiple temporal encoders 20 may be provided for U-Net encoders 11, 12, 13 and U-Net decoder 30, respectively.

[0043] In a preferred embodiment, the style adaptive neural network 100 further includes zero convolution modules 21, 23. The output of the zero convolution module 21 is connected to the input of the adder 50, and the zero convolution module 23 is connected between the output of the U-Net encoder 13 and a second input of the U-Net decoder 30.

[0044] In another preferred embodiment, the style adaptive neural network 100 further includes a zero convolution module 22. The output of the zero convolution module 22 is connected to the input of the adder 50.

[0045] During the training of the style adaptive neural network 100, we first add noise to the original image to obtain a noisy input image 1, and then calculate the latent variable z t Obtain the latent variable z t is input to the U-Net encoder 11, the adder 40, and the adder 50. A line drawing image C f is input to the adder 40, and after being zero-folded by the zero-folding module 21, is input to the adder 50. t and line drawing image C f The latent variable z t and the zero-convolved line drawing image C f are added by an adder 50 and then input to the U-Net encoder 13. The outputs of the U-Net encoders 11 and 12 are provided to the first input and the second input of the U-Net decoder 30, respectively, and the output of the U-Net encoder 13 is zero-convolved by a zero-convolution module 23 and then provided to the second input of the U-Net decoder 30.

[0046] During training, in order for the style adaptive neural network 100 to converge, only the parameters of the U-Net encoder 13 may be adjusted, while the parameters of the U-Net decoder 30, the text encoder 10, the temporal encoder 20, and the U-Net encoders 11 and 12 may be kept unchanged.

[0047] In a preferred embodiment, during training, the parameters of the U-Net Encoder 13 and the U-Net Decoder 30 may be adjusted while the parameters of the text encoder 10, the temporal encoder 20, and the U-Net encoders 11 and 12 may be kept unchanged so that the style adaptive neural network 100 converges.

[0048] The parameters include weights and biases.

[0049] In a preferred embodiment, during training, the style reference image 2' may be provided to the adder 50 after being zero-convolved by the zero-convolution module 22. The style reference image 2' is obtained after performing a TPS transformation on the style reference image 2. In this case, the adder 50 adds the latent variable z t and the zero-convolved line drawing image C f and the style reference image 2′ that has been zero-convolved, and the result of the addition is provided to the U-Net encoder 13.

[0050] Note that during training, the style reference image 2 corresponds to the original image of the noisy input image 1.

[0051] It should be noted that the zero convolution modules 21, 22 and 23 mean that their weights are zero only for the first iteration for training the style adaptive neural network 100, and are non-zero during subsequent iterations of training and during inference.

[0052] Also, line drawing image C fis encoded by an encoder (not shown) that participates in training the neural network before being input to the adder 40 and the zero convolution module 21. The style reference image 2′ has its latent variables C s The above encoding of the line drawing image and the style reference image may use any suitable conventional technique, the description of which is omitted here.

[0053] Note that we can use conventional DDPM to add noise to the original image to obtain the noisy input image 1, or we can use a pre-trained autoencoder in LDM to map the original image to the latent variables z t Then, noise may be added to the latent variables by DDPM. In order to facilitate understanding of the present disclosure, the following briefly introduces the process of adding noise to the original image and obtaining the latent variables of the image with added noise.

[0054] FIG. 2 is a schematic diagram showing the forward and reverse processes of DDPM. As shown in FIG. 2, in the forward process from right to left, X0 to X T We continuously add noise to the image up to and diffuse it to obtain the noise-added image. In the reverse process from left to right, we get the noise-added image. T The noise is removed stepwise from the image from X to X0 to obtain a noise-free image. The forward and backward processes may occur in pixel space or in latent space.

[0055] In the embodiment of the present disclosure, the DDPM-based LDM is used to perform the processes of forward diffusion and backward denoising in the latent space.

[0056] 3 is a schematic diagram showing an LDM 300. As shown in FIG. 3, the LDM 300 includes a latent space 301, a condition module 302, a diffusion module 303, a denoising module 304, and a pixel space 305.

[0057] First, the original image x0 is mapped to the latent space 301 by the pre-trained encoder to obtain the latent variable z0, and then the diffusion module 303 performs a forward diffusion process to obtain z T Get z T , and provide the new tensor to the denoising module 304 for the reverse denoising process. In the reverse process, the final noise-free latent variable z0 is obtained, and then the pre-trained decoder maps it back to the pixel space 305 to obtain the noise-free image x0. The condition module 302 is used to provide the denoising module 304 with conditions of types such as image, text and semantic maps.

[0058] Two types of diffusion processes can be performed in the LDM 300:

[0059] In the first type of diffusion process, we first provide a noise-free latent variable z0, and then perform the diffusion process step by step. After t steps of diffusion are performed, we obtain the noisy latent variable z t can be obtained.

[0060]

number

[0061] Preferably, in the second type of diffusion process, the derivation of the DDPM reparameterization is used to directly go from z0 to z by sampling the noise only once. t may be obtained.

[0062]

number

[0063] The above is the latent variable z of the input of the style adaptive neural network 100. t We have briefly described the generation process of L. The following describes the loss function L of the style adaptation neural network 100.

[0064] If no style reference image 2′ is provided, the loss function of the style adaptation neural network 100 is:

[0065]

number

[0066] Noise at z0 (Outside 10) Adding TIFF2025011048000014.tif15170 results in z t That is, (Outside 11) TIFF2025011048000015.tif19170, where (Outside 12) TIFF2025011048000016.tif16170 is a preset constant. Noise (Outside 13) TIFF2025011048000017.tif15170 is used as a label (ground truth) and is set as a target to be predicted by the style adaptive neural network 100. The network parameter to be optimized in the style adaptive neural network 100 is θ. As described above, the network parameter θ to be optimized is related to the U-Net encoder 13, and preferably also to the U-Net decoder 30.

[0067] During training, the style adaptive neural network is t , t, c t , c f Enter z t Noise contained in (Outside 14) Predict TIFF2025011048000018.tif17170. And the loss function L of the style adaptation neural network 100 is (Outside 15) TIFF2025011048000019.tif12170 and (Outside 16) Calculate the L2 loss (squared loss function) for TIFF2025011048000020.tif17170.

[0068] Given a style reference image 2′, the loss function L′ of the style adaptation neural network 100 is:

[0069]

number

[0070] During the training period, s The original image is converted to a style reference image after transformation. (Outside 17) Get TIFF2025011048000022.tif19170 and encode it as c s During the inference period, c s may be obtained by inputting any image to the encoder, which may be unrelated or related to the original image.

[0071] Note that optimizing the network parameter θ is a function of the gradient of the network parameter according to the loss function L or L′. (Outside 18) Calculate TIFF2025011048000023.tif23170 (where (Outside 19) TIFF2025011048000024.tif19170) and the parameters according to the random gradient descent algorithm (outside 20) Update TIFF2025011048000025.tif17170 (i.e. (outside 21) TIFF2025011048000026.tif25170), where μ is a preset learning rate.

[0072] By optimizing the network parameters, the style adaptive neural network 100 is trained until it converges (i.e., the loss function does not decrease any more) or until a preset number of training rounds is reached.

[0073] Note that before training the style adaptive neural network 100, the U-Net encoders 11 and 12 and the U-Net decoder 30 are pre-trained.

[0074] During inference of the style adaptive neural network 100, the prompt “Gauguin style”, the line drawing image and latent variables corresponding to the original image, and the style reference image can be input into the trained style adaptive neural network, and a color image that matches the line drawing prompt and the style reference image can be obtained at the output of the U-Net decoder 30.

[0075] Note that the content of the prompt is not limited to image style, but may include any text that describes the content to be generated.

[0076] Also, line drawings may be provided that are extracted from the original image, or may be obtained directly.

[0077] Also, during inference, the style reference image may be any image that is unrelated or related to the original image.

[0078] In addition, the style adaptive neural network 100 of FIG. 2 further includes a Basic Transformer Block (not shown), which may be considered to be included within the U-Net decoder 30.

[0079] The prompt (also called suggestion) control alone can generate an artist's style, but an artist can have multiple substyles. However, substyles cannot be precisely described in words. Therefore, the prompts only generate a random one of the multiple substyles. By inputting additional style-reference images, an image with the desired substyle can be better generated.

[0080] The following describes a method 400 for training a neural network according to an embodiment of the present disclosure, shown in FIG. 4, with reference to the style adaptive neural network 100 shown in FIG.

[0081] First, in step 401, different noises are added to the original image to obtain an input image. Specifically, in this embodiment, the original image is converted into a noise-added input image 1 using the LDM 300 shown in FIG. 3, and then the latent variable z t Get the.

[0082] Next, in step 402, the input image is input to a first U-Net encoder, and the line drawing image corresponding to the original image and the input image are added by a first adder, and then input to a second U-Net encoder. Specifically, in this embodiment, the latent variable z t is input to the U-Net encoder 11, and a line drawing image C f and the latent variable z t are added by an adder 40 and then input to the U-Net encoder 12.

[0083] Next, in step 403, the input image and the line drawing image are input to a second adder. t and the encoded line drawing image C f and are input to an adder 50.

[0084] Preferably, the encoded line drawing image C fmay be zero-convolved by zero-convolution module 21 and provided to adder 50.

[0085] Next, in step 404, the second adder inputs the sum of the input image and the line drawing image to the third U-Net encoder. Specifically, in this embodiment, the adder 50 adds the latent variable z t and the encoded line drawing image C f The sum of these is input to the U-Net encoder 13.

[0086] Finally, in step 405, the neural network is trained. Specifically, in this embodiment, when training the style adaptive neural network 100, only the parameters of the U-Net encoder 13 may be adjusted so that the style adaptive neural network 100 converges, while the parameters of the U-Net decoder 30, the text encoder 10, the time encoder 20, and the U-Net encoders 11 and 12 may be maintained as they are.

[0087] Preferably, when training the style adaptive neural network 100, the parameters of the U-Net encoder 13 and the U-Net decoder 30 may be adjusted while keeping the parameters of the text encoder 10, the temporal encoder 20, and the U-Net encoders 11 and 12 unchanged so that the style adaptive neural network 100 converges.

[0088] In this embodiment, the loss function L of the style adaptive neural network 100 is shown in equation (3) above.

[0089] In one preferred embodiment, in step 403′, the style reference image may be input to the second adder. Specifically, in this embodiment, the latent variable C obtained by a pre-trained encoder (not shown) of the style reference image 2′ is s is also input to the adder 50, which calculates the latent variable z t , the encoded line drawing image C fand the latent variable C of style reference image 2' s The sum of these is input to the U-Net encoder 13.

[0090] In the preferred embodiment, the loss function L' of the style adaptive neural network 100 is shown in equation (4) above.

[0091] Preferably, the latent variables C of the style reference image 2′ s may also be provided to summer 50 after being zero-convolved by zero-convolution module 22 .

[0092] The style reference image 2' is obtained by subjecting the style reference image 2 to TPS conversion.

[0093] During training, the style reference image 2 is a set of noise-added input image 1 and wire manuscript C f During inference, the style reference image 2 corresponds to the original image of the noisy input image 1 and the line drawing C f The image may be any image that may or may not be related to the original image.

[0094] Preferably, during training, the output result of the U-Net encoder 13 may be input to the second input of the U-Net decoder 30 after being zero-convolved by the zero-convolution module 23.

[0095] The method 400 for training a neural network, shown in FIG. 4, can generate high quality color images with a desired style.

[0096] FIG. 5 illustrates diagrammatically the image quality of the output obtained using the trained style adaptive neural network 100 shown in FIG. 1 after training the style adaptive neural network 100 using the method 400 of FIG.

[0097] 5, for example, the prompt input (not shown) to the text encoder 10 is "Gauguin style" and the style-referential image 2' corresponds to a sub-style of Gauguin. t and line drawing image C f , as well as a style reference image 2' having a Gauguin style, the style of the image output by the U-Net decoder 30 matches well with the style reference image 2'.

[0098] The above method may be fully implemented by a computer executable program, or may be partially or fully implemented using hardware and / or firmware. When implemented by hardware and / or firmware, or when the computer executable program is loaded into a hardware device capable of executing the program, a neural network training device described below is realized. The following will omit the above detailed content and provide an overview of these devices. Note that these devices can execute the above method, but the method is not limited to adopting or being executed by the components of the devices described below.

[0099] 6 is a block diagram of a neural network training device 600 according to an embodiment of the present disclosure. The device 600 includes an acquisition unit 601, a first input unit 602, a second input unit 603, and The neural network includes a training unit 604. The acquisition unit 601 acquires an input image by adding noise to an original image. The first input unit 602 inputs the input image to a first U-Net encoder, and adds a line drawing image corresponding to the original image and an input image by a first adder, and then inputs the result to a second U-Net encoder. The second input unit 603 adds the input image and the line drawing image by a second adder, and then inputs the result to a third U-Net encoder. The training unit 604 trains the neural network. When training the neural network, the training unit 604 adjusts parameters of the third U-Net encoder so that the neural network converges, and keeps the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged, or adjusts parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges, and keeps the parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged.

[0100] The neural network training device 600 shown in Fig. 6 corresponds to the neural network training method 400 shown in Fig. 4. Therefore, the details of each part of the neural network training device 600 have already been explained in detail in the explanation of the neural network training method 400 in Fig. 4, and therefore the explanation thereof will be omitted here.

[0101] The above-mentioned processes and devices may be realized by software and / or firmware. When implemented by software and / or firmware, a program constituting software for implementing the above-mentioned method may be installed from a storage medium or a network into a computer having a dedicated hardware configuration (e.g., a general-purpose personal computer 700 shown in FIG. 7), and the computer can execute various functions when various programs are installed therein.

[0102] 7 is a block diagram showing an exemplary configuration of a general-purpose personal computer capable of implementing a method and / or apparatus according to an embodiment of the present disclosure. As shown in FIG. 7, a central processing unit (CPU) 701 executes various processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage unit 708 to a random access memory (RAM) 703. The RAM 703 stores data necessary for the CPU 701 to execute various processes as necessary. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output interface 705 is also connected to the bus 704.

[0103] An input unit 706 (including a keyboard, a mouse, etc.), an output unit 707 (including a display, such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage unit 708 (including, for example, a hard disk, etc.), and a communication unit 709 (including, for example, a network interface card, such as a LAN card, a modem, etc.) are connected to the input / output interface 705. The communication unit 709 executes communication processing via a network, such as the Internet. If necessary, a driver 710 may be connected to the input / output interface 705. The removable medium 711 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., and is set up in the driver 710 as necessary, and a computer program read out from the medium is installed in the storage unit 708 as necessary.

[0104] When the above processes are performed by software, a program constituting the software is installed via a network, such as the Internet, or a storage medium, such as a removable medium 711 .

[0105] It should be noted that these storage media are not limited to the removable medium 711 shown in Fig. 7 that stores the program and provides the program to the user separately from the device. The removable medium 711 includes, for example, a magnetic disk (including a floppy disk (registered trademark)), an optical disk (including an optical disk-read only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto-optical disk (a mini disk (MD) (registered trademark)), and a semiconductor memory. Alternatively, the storage medium may be a ROM 702, a hard disk included in the storage unit 708, or the like, which stores the program and is provided to the user together with the device including them.

[0106] The present disclosure further provides a computer program product having stored thereon corresponding computer program code, machine readable instruction code, which when read and executed by a machine, is capable of performing the method according to the embodiments of the present disclosure described above.

[0107] Accordingly, the present disclosure further includes a storage medium having recorded thereon a program product including machine-readable instruction code, including, but not limited to, a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0108] Furthermore, the following supplementary notes are disclosed regarding the embodiments including the above-mentioned examples. (Appendix 1) 1. A method for training a neural network, the neural network comprising: a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder; The method comprises: adding noise to an original image to obtain an input image; inputting the input image to the first U-Net encoder, adding a line drawing image corresponding to the original image and the input image by the first adder, and then inputting the resulting sum to the second U-Net encoder; adding the input image and the line drawing image by the second adder and then inputting the resulting image to the third U-Net encoder; training the neural network; When training the neural network, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged; or adjusting parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges, and keeping parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged. (Appendix 2) the text encoder and the time encoder are connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder, and the U-Net decoder; The method comprises: inputting a prompt to the text encoder and a time step to the temporal encoder to obtain tensors for the prompt and the time step; 2. The method of claim 1, further comprising the step of providing tensors for the prompt and each of the time steps to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder, and the U-Net decoder, respectively. (Appendix 3) The neural network further includes a first zero-convolution module and a second zero-convolution module, the first zero-convolution module being connected to the second adder, and the second zero-convolution module being connected between an output of the third U-Net encoder and a second input of the U-Net decoder; 3. The method of claim 2, wherein the line drawing image is zero-convolved by the first zero-convolution module, and the input image and the zero-convolved line drawing image are added by the second adder before being input to the third U-Net encoder. (Appendix 4) the neural network further includes a third zero-convolution module, the third zero-convolution module coupled to the second adder; The method comprises: 4. The method of claim 3, further comprising the step of zero-convolving the style reference image with the third zero-convolution module and inputting the zero-convolved style reference image to the second adder. (Appendix 5) 5. The method of claim 4, wherein during training, the style reference images are obtained by performing a thin plate spline transformation on the original images. (Appendix 6) 5. The method of claim 4, wherein during inference, the style reference image is any other image different from the original image. (Appendix 7) 7. The method of any of claims 1 to 6, wherein latent variables of the input image are obtained by a pre-trained encoder before the input image is input to the first U-Net encoder, the first adder, and the second adder. (Appendix 8) 5. The method of claim 4, wherein the line drawing image is encoded by an encoder that participates in training the neural network before being input to the first adder and the first zero convolution module. (Appendix 9) 7. The method of claim 5 or 6, wherein latent variables of the style reference image are obtained by a pre-trained encoder before the style reference image is input to the third zero convolution module. (Appendix 10) 7. The method of claim 1, wherein the neural network further includes a basic transformation block included in the U-Net decoder. (Appendix 11) the neural network is a style adaptive neural network; 7. The method of any of claims 1 to 6, wherein the style includes one or more of color, brightness, saturation, contrast, texture, and line of an image. (Appendix 12) the first U-Net encoder, the second U-Net encoder, and the U-Net decoder are pre-trained prior to training the neural network; The method of any one of claims 1 to 6, further comprising replicating network parameters obtained by pre-training the second U-Net encoder as initial parameters of a third U-Net encoder. (Appendix 13) An apparatus for training a neural network, the neural network comprising: a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and the output of the second U-Net encoder and the output of the third U-Net encoder are connected to a second input of the U-Net decoder; The apparatus comprises: an acquisition unit that acquires an input image by adding noise to an original image; A first input unit that inputs the input image to the first U-Net encoder, adds a line drawing image corresponding to the original image and the input image by the first adder, and then inputs the result to the second U-Net encoder; A second input unit that adds the input image and the line drawing image by the second adder and then inputs the result to the third U-Net encoder; a training unit for training the neural network; The training unit, when training the neural network, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged; or The apparatus adjusts parameters of the U-Net decoder and the third U-Net encoder such that the neural network converges, and keeps parameters of the text encoder, the temporal encoder, the first U-Net encoder, and the second U-Net encoder unchanged. (Appendix 14) A computer-readable storage medium having a program stored thereon, the program causing a computer to perform a method according to any one of claims 1 to 12 when executed by a processor. (Appendix 15) A neural network comprising: a text encoder into which a prompt is input to obtain a tensor of the prompt; a time encoder into which the time steps are input so as to obtain a tensor of said time steps; A first U-Net encoder, a second U-Net encoder, and a third U-Net encoder; a U-Net decoder, wherein an output of the first U-Net encoder is connected to a first input of the U-Net decoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder; a first adder connected to an input of the second U-Net encoder and a second adder connected to an input of the third U-Net encoder; a first zero-convolution module connected to the second adder, and a second zero-convolution module connected between an output of the third U-Net encoder and a second input of the U-Net decoder; the text encoder and the time encoder are connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder, and the U-Net decoder; As the neural network is trained, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged; or A neural network comprising: a first U-Net encoder and a second U-Net encoder; a first U-Net encoder and a second U-Net encoder; and a second U-Net encoder, each of which has a different parameter; and a second U-Net encoder and a second U-Net encoder. (Appendix 16) 16. The neural network of claim 15, further comprising a third zero-convolution module, the third zero-convolution module coupled to the second summer. (Appendix 17) Add noise to the original image to get the input image, During the training period, The input image is input to the first U-Net encoder; A line drawing image corresponding to the original image and the input image are added by the first adder, and then input to the second U-Net encoder; performing zero convolution on the line drawing image by the first zero convolution module, and performing convolution on the first style reference image by the third zero convolution module; The zero-convolved line drawing image, the zero-convolved first style reference image, and the input image are input to the second adder and added, and then input to the third U-Net encoder; During the inference period, The input image is input to the first U-Net encoder; The line drawing image and the input image are added by the first adder, and then input to the second U-Net encoder; performing zero convolution on the line drawing image by the first zero convolution module, and performing convolution on a second style reference image by the third zero convolution module; 17. The neural network of claim 16, wherein a zero-convolved line drawing image, a zero-convolved second style reference image, and the input image are input to the second adder and added together, and then input to the third U-Net encoder. (Appendix 18) the first style reference image is obtained by performing a thin plate spline transformation on the original image; 18. The neural network of claim 17, wherein the second style reference image is any other image different from the original image. (Appendix 19) 19. The neural network of any of claims 15 to 18, further comprising a basic transform block included in the U-Net decoder. (Appendix 20) 19. The neural network of any of claims 15 to 18, wherein the neural network is a style adaptive neural network.

[0109] It should be noted that the terms "comprise," "have," or any other variation thereof are not limited to exclusive inclusion, and a process, method, article, or apparatus that includes a set of elements includes not only those elements, but also other elements not expressly listed or inherent elements of the process, method, article, or apparatus. Also, unless further limited, a more limited term "comprises a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0110] The above describes the preferred embodiments of the present disclosure with reference to the drawings, but the above embodiments and examples are illustrative and not restrictive. Those skilled in the art may make various modifications, improvements, and equivalent changes to the present disclosure within the spirit and scope of the claims. These modifications, improvements, and equivalent changes are included in the protection scope of the present disclosure.

Claims

1. A method for training a neural network, the neural network comprising: a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and the output of the second U-Net encoder and the output of the third U-Net encoder are connected to a second input of the U-Net decoder; The method comprises: adding noise to an original image to obtain an input image; inputting the input image to the first U-Net encoder, adding the line drawing image corresponding to the original image and the input image by the first adder, and then inputting the result to the second U-Net encoder; adding the input image and the line drawing image by the second adder and then inputting the resulting image to the third U-Net encoder; training the neural network; When training the neural network, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged, or A method comprising: adjusting parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges; and maintaining parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged.

2. the text encoder and the time encoder are connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder and the U-Net decoder; The method comprises: inputting a prompt to the text encoder and a time step to the temporal encoder to obtain tensors for the prompt and the time step; and providing tensors for each of the prompts and the time steps to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder, and the U-Net decoder, respectively.

3. The neural network further includes a first zero-convolution module and a second zero-convolution module, the first zero-convolution module being connected to the second adder, and the second zero-convolution module being connected between an output of the third U-Net encoder and a second input of the U-Net decoder; 3. The method of claim 2, wherein the line drawing image is zero-convolved by the first zero-convolution module, and the input image and the zero-convolved line drawing image are added by the second adder before being input to the third U-Net encoder.

4. the neural network further includes a third zero-convolution module, the third zero-convolution module coupled to the second adder; The method comprises:

4. The method of claim 3, further comprising the steps of: zero-convolving the style reference image with the third zero-convolution module; and inputting the zero-convolved style reference image to the second adder.

5. The method of claim 4 , wherein during training, the style reference images are obtained by performing a thin-plate spline transformation on the original images.

6. The method of claim 4 , wherein during inference, the style reference image is any other image different from the original image.

7. The method of claim 1 , wherein the first U-Net encoder, the second U-Net encoder and the U-Net decoder are pre-trained before training the neural network.

8. An apparatus for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-Net encoder, a second U-Net encoder, a third U-Net encoder, a U-Net decoder, a first adder, and a second adder, an output of the first U-Net encoder is connected to a first input of the U-Net decoder, the first adder is connected to an input of the second U-Net encoder, the second adder is connected to an input of the third U-Net encoder, and the output of the second U-Net encoder and the output of the third U-Net encoder are connected to a second input of the U-Net decoder; The apparatus comprises: an acquisition unit that acquires an input image by adding noise to an original image; a first input unit that inputs the input image to the first U-Net encoder, adds a line drawing image corresponding to the original image and the input image by the first adder, and then inputs the result to the second U-Net encoder; a second input unit that adds the input image and the line drawing image by the second adder and then inputs the resultant to the third U-Net encoder; a training unit for training the neural network; The training unit, when training the neural network, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged, or The apparatus adjusts the parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges, and keeps the parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged.

9. A neural network comprising: a text encoder into which a prompt is input to obtain a tensor of the prompt; a time encoder into which the time steps are input so as to obtain a tensor of said time steps; a first U-Net encoder, a second U-Net encoder and a third U-Net encoder; a U-Net decoder, wherein an output of the first U-Net encoder is connected to a first input of the U-Net decoder, and an output of the second U-Net encoder and an output of the third U-Net encoder are connected to a second input of the U-Net decoder; a first adder connected to an input of the second U-Net encoder, and a second adder connected to an input of the third U-Net encoder; a first zero-convolution module connected to the second adder, and a second zero-convolution module connected between an output of the third U-Net encoder and a second input of the U-Net decoder; the text encoder and the time encoder are connected to the first U-Net encoder, the second U-Net encoder, the third U-Net encoder and the U-Net decoder; As the neural network is trained, Adjusting parameters of the third U-Net encoder so that the neural network converges, and keeping the parameters of the U-Net decoder, the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged, or A neural network, comprising: adjusting parameters of the U-Net decoder and the third U-Net encoder so that the neural network converges; and maintaining parameters of the text encoder, the time encoder, the first U-Net encoder, and the second U-Net encoder unchanged.

10. 10. The neural network of claim 9, further comprising a third zero-convolution module, said third zero-convolution module coupled to said second adder.

Images