Neural network training device, inference device, and computer program

Abstract

To provide a neural network training device, an inference device, and a computer program.SOLUTION: A neural network includes a text encoder, a time encoder, first to fourth UNET encoders, an UNET decoder, and first to third adders. A training unit includes: a unit which acquires a noise image by adding different noises to an original image; a unit which inputs the noise image to the first UNET encoder, adds a line drawing image corresponding to the original image and the noise image and then inputs the addition results into the second and third UNET encoders, and adds the noise image and an image which corresponds to the original image and a part of which is masked and then inputs the addition results to the fourth UNET encoder; and a unit which adjusts parameters of the third and fourth UNET encoders, and converges the neural network by making the parameters of each of the text encoder, the time encoder, and the first and second UNET encoders constant.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of neural networks, and in particular to the rendering of line art images using neural networks. [Background technology]

[0002] Coloring line drawings is an essential part of the anime and manga production process, but it is difficult to ensure both quality and continuity when rendering long horizontal or vertical lines.

[0003] So far, two methods have been proposed to improve the continuity between adjacent image blocks while maintaining the quality of the rendered image.

[0004] The first method involves splitting a long line drawing into multiple blocks, rendering each block separately, and then stitching the rendered blocks together. This solves the problem of large GPU memory requirements, but the stitched image has a lower continuity because there are obvious seams between adjacent blocks.

[0005] The second method involves first rendering a scaled-down version of the original long line drawing, and then using a ControlNet Tile model to improve the detail of the rendered image of the scaled-down version of the line drawing, thus improving the continuity of the image, but changing the content and detail within the line drawing. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide a neural network training device, an inference device and a computer program. [Means for solving the problem]

[0007] According to one aspect of the present invention, there is provided an apparatus for training a neural network, the neural network including a text encoder, a time encoder, a first UNET (Unet) encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and the output end of the second UNET encoder, the output end of the third UNET encoder and the output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, the apparatus comprising: an obtaining unit configured to obtain a noise image by adding different noises to the original image; an input unit configured to input a noise image to a first UNET encoder, add a line drawing image corresponding to an original image and a noise image by a first adder, and then input the result to a second UNET encoder, add the noise image and the line drawing image by a second adder, and then input the result to a third UNET encoder, and add the noise image and a partially masked (covered) image (mask image) corresponding to the original image by a third adder, and then input the result to a fourth UNET encoder; and Including training units The training unit is configured to train the neural network in the following manner: Adjusting parameters of the third UNET encoder and the fourth UNET encoder, and keeping parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, thereby allowing the neural network to converge; or The parameters of the UNET decoder, the third UNET encoder, and the fourth UNET encoder are adjusted, and the parameters of the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder are kept unchanged, allowing the neural network to converge.

[0008] Preferably, the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder. A prompt is input to the text encoder, and a time step is input to the time encoder to obtain tensors of the prompt and the time step. The tensors of the prompt and the time step are provided to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder, respectively.

[0009] Preferably, the neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output terminal of the third UNET encoder and the second input terminal of the UNET decoder. The line drawing image is provided to the second adder after being zero-convolved by the first zero convolution module.

[0010] Preferably, the training unit is further configured to adjust parameters of the first zero convolution module and parameters of the second zero convolution module during training.

[0011] Preferably, the partially masked image is obtained by randomly masking parts of the original image with solid colors.

[0012] Preferably, before the noise image is input to the first UNET encoder, the first adder, the second adder and the third adder, the hidden variables of the original image are obtained by another pre-trained encoder, and then the hidden variables corresponding to the noise image are obtained by adding noise to the hidden variables.

[0013] Preferably, the line drawing image is coded by another coder which participates in training the neural network before being input to the first adder and the first zero convolution module.

[0014] Preferably, the partially masked image is coded by another coder which takes part in training the neural network before being input to the third adder.

[0015] Preferably, the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder are pre-trained before training the neural network.

[0016] Preferably, the total loss function of the neural network is constructed as a sum of a first loss function and a second loss function, in which the first loss function is a sum-of-squares loss function between the randomly sampled noise and the noise estimated for the noisy image, and the second loss function is a sum-of-squares loss function between the hidden variables corresponding to the original noise-free image and the hidden variables of the reconstructed image after noise removal.

[0017] According to another aspect of the present invention, there is provided an apparatus for performing inference using a pre-trained neural network as described above, the apparatus comprising: an image segmentation unit configured to segment the line drawing image into N image blocks, where adjacent image blocks among the N image blocks have overlapping portions, where N is an integer greater than or equal to 2; and Including an inference unit, The inference unit is adapted to perform the following operations: A first noise image among the N noise images is input to a first UNET encoder, the first noise image and a first image block among the N image blocks of the line drawing image are added by a first adder and a second adder, respectively, and then input to a second UNET encoder and a third UNET encoder, respectively, the first noise image and a pure color image are added by a third adder, and then input to a fourth UNET encoder; a first mask image block is obtained based on a first output image block corresponding to the first image block of the line drawing image output from the UNET decoder; the second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image among the N noise images by the first adder and the second adder, respectively, and then the results are respectively output to a second UNET encoder. the first mask image block to the N-1th mask image block are input to a third adder and a third UNET encoder, the first mask image block to the N-1th mask image block are sequentially added to the second noise image to the Nth noise image by a third adder, and the result is then input to a fourth UNET encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image, among which, portions of the first output image block to the N-1th output image block output from the UNET encoder that overlap with the second image block to the Nth image block of the line drawing image are extracted respectively to form parts of the first to N-1th mask image blocks, and other portions of the first to N-1th mask image blocks are filled (filled) with a solid color to obtain the first mask image block to the N-1th mask image block.

[0018] Preferably, the apparatus for inference further includes a stitching unit, which is configured to stitch together the N output image blocks output from the UNET decoder in sequence to form a complete image.

[0019] Preferably, the stitching unit is further configured to stitch the N output image blocks in a weighted fusion manner, such that parts of the N output image blocks closer to an edge have higher weights.

[0020] Preferably, before any image is input to the first UNET encoder, the first adder, the second adder and the third adder to obtain N noise images, the hidden variables of the any image are obtained by another pre-trained encoder, and then different noises, for example Gaussian noise, are added to the hidden variables to obtain N noise images.

[0021] Preferably, the N image blocks of the line drawing image are encoded by other encoders participating in the training of the neural network before being input to the first adder and the first zero convolution module, among which the first zero convolution module and the second zero convolution module have parameters that do not become zero during the inference period of the neural network.

[0022] Preferably, before the N-1 mask image blocks are input to the third adder, they are encoded by another encoder participating in the training of the neural network to obtain hidden variables for the N-1 mask image blocks.

[0023] Preferably, the colors for the solid color image and the masks of the 1st to N-1th mask image blocks are the same as the colors for masking the image when training the neural network.

[0024] Preferably, the sizes of the N image blocks of the line art image, the N output image blocks and the solid color image are the same.

[0025] According to other aspects of the invention, there are further provided corresponding computer program code, a computer readable storage medium and a computer program product. Effect of the Invention

[0026] The device for training neural networks and the device for inference with pre-trained neural networks of the present invention allow for improved continuity in rendered images, thereby improving the quality of the rendered images and also preserving content and details in line drawings. [Brief description of the drawings]

[0027] [Figure 1] FIG. 2 illustrates a neural network according to an embodiment of the present invention. [Diagram 2] FIG. 1 illustrates the forward and backward processes of the DENOISING DIFFUSION PROBABILITY MODEL (DDPM). [Diagram 3] FIG. 1 is a diagram showing the latent diffusion model (LDM, LATENT DIFFUSION MODEL). [Figure 4] 2 is a flowchart of a method for training a neural network in an embodiment of the present invention. [Diagram 5] 1 is a flowchart of a method for performing inference using a pre-trained neural network in an embodiment of the present invention. [Figure 6] FIG. 6 illustrates image segmentation and masking in the method of FIG. 5. [Figure 7] FIG. 6 illustrates weighted fusion of image stitching in the method of FIG. 5. [Figure 8] FIG. 2 is a block diagram of an apparatus for training a neural network in an embodiment of the present invention. [Figure 9] FIG. 1 is a block diagram of an apparatus for performing inference using a pre-trained neural network in an embodiment of the present invention. [Figure 10] FIG. 2 is a block diagram of an exemplary configuration of a general-purpose personal computer in which methods and / or apparatus according to embodiments of the present invention may be implemented. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments are merely illustrative and are not intended to limit the present invention.

[0029] To solve the problems of the prior art, the present invention proposes a new method of improving continuity. First, the ControlNet is extended to support multiple controls, including line drawing control and rendering control, and the previously rendered image block is given as a prompt to improve the continuity between adjacent image blocks, thereby improving the rendering control and improving the image quality. Second, the continuity between image blocks is further improved by weighted fusion.

[0030] The neural network of the present invention is based on a pre-trained model, e.g., ControlNet V1.1 Anime Lineart, and utilizes the image knowledge of the pre-trained model, uses different neural network branches to control the line art and rendering respectively, and freezes the weights of the pre-trained model to avoid the catastrophic forgetting problem.

[0031] FIG 1 illustrates a neural network 100 according to an embodiment of the present invention. As shown in FIG 1, the neural network 100 includes UNET encoders 11, 12, 13, 14, a text encoder 10, a time encoder 20, a UNET decoder 21, and adders 22, 23, 24. The output of the UNET encoder 11 is connected to a first input of the UNET decoder 21. The output of the adder 22 is connected to an input of the UNET encoder 12, the output of the adder 23 is connected to an input of the UNET encoder 13, and the output of the adder 24 is connected to an input of the UNET encoder 14. The outputs of the UNET encoders 12, 13, 14 are connected to a second input of the UNET decoder 21. The output of the UNET decoder 21 provides an inference result of the neural network 100.

[0032] It should be noted that the text encoder 10 and the time encoder 20 are connected to each of the UNET encoders 11, 12, 13, 14 and the UNET decoder 21.

[0033] As is known, prompts are input to a text encoder 10 to obtain a prompt tensor, and time steps are input to a time encoder 20 to obtain a time step tensor. The prompt and time step tensors are respectively provided to each of the UNET encoders 11, 12, 13, 14 and the UNET decoder 21. It is noted that the prompt content may include various characteristics of the image that needs to be rendered.

[0034] Also, although FIG. 1 shows one text encoder 10 and one time encoder 20, the present invention is not limited to this, and multiple text encoders 10 and multiple time encoders 20 may be provided for each of the UNET encoders 11, 12, 13, 14 and UNET decoder 21, respectively.

[0035] According to one preferred embodiment, neural network 100 further includes zero-convolution modules 25, 26. An output of zero-convolution module 25 is connected to an input of summer 23, and zero-convolution module 26 is connected between an output of UNET encoder 13 and a second input of UNET decoder 21.

[0036] During the training of the neural network 100, we first obtain a noisy image by adding noise to the original image, and then we calculate the hidden variable Z t We need to obtain the hidden variable Z t are input to the UNET encoder 11 and adders 22, 23, and 24. A line drawing image corresponding to the original image is input to the adder 22, and is zero-convolved by the zero-convolution module 25 before being input to the adder 23. The hidden variable Z t The line drawing image is added by an adder 21 and then input to the UNET encoder 12. t The line drawing image after zero convolution is added by an adder 23 and then input to the unet encoder 13. tand the partially masked original image are added by adder 24 and then input to UNET encoder 14. The output of UNET encoder 11 is provided to a first input of UNET decoder 21, the outputs of UNET encoders 12 and 14 are provided to a second input of UNET decoder 21, and the output of UNET encoder 13 is zero-convolved by zero-convolution module 26 and then provided to a second input of UNET decoder 21.

[0037] During training, the neural network 100 is allowed to converge by adjusting the parameters of only the UNET encoders 13, 14 and the zero convolution modules 25, 26, while keeping the parameters of the UNET decoder 21, the text encoder 10, the time encoder 20, and the UNET encoders 11, 12 unchanged.

[0038] Optionally, during training, the parameters of the UNET encoders 13, 14, the zero convolution modules 25, 26, and the UNET decoder 21 may be adjusted while keeping the parameters of the text encoder 10, the time encoder 20, and the UNET encoders 11, 12 unchanged, allowing the neural network 100 to converge.

[0039] It should be noted that the above parameters include weights and deviations.

[0040] Note that during the training phase, the mask image may be obtained by masking a portion of the original image, for example any one of the top, bottom, left or right portions, with a solid color, preferably black.

[0041] It is noted that zero-convolution modules 25, 26 are convolution modules whose weights are zero during the first iteration of training neural network 100 and are all non-zero during subsequent iterations and inference.

[0042] In addition, the line drawing image is encoded by an encoder (not shown) that participates in neural network training before being input to the adder 22 and the zero convolution module 25, and the mask image has its features acquired by a pre-trained encoder (not shown) before being input to the adder 24. The encoding of the line drawing image and the mask image may be performed by any suitable conventional technique, which will not be described in detail here.

[0043] Furthermore, we can obtain a noisy image by adding noise to the original image using the existing DDPM, and we can also use a pre-trained autoencoder in the LDM to map the original image to the hidden variables Z t Then, noise may be added to the hidden variables by DDPM. In order to easily understand the present invention, the process of adding noise to the original image and obtaining the hidden variables of the image after adding noise is briefly introduced below.

[0044] FIG. 2 shows the forward and reverse processes of DDPM. As shown in FIG. 2, in the forward process from right to left, X0 to X t By continuously adding noise to the image and performing diffusion, we obtain a noisy image. In the reverse process from left to right, t By progressively removing noise from a noisy image from X0 to X0, a noise-free image is obtained. The forward and backward processes may occur in pixel space or in hidden space.

[0045] According to an embodiment of the present invention, a DDPM-based LDM is used to perform forward diffusion and backward denoising processes in the hidden space.

[0046] 3 is a block diagram illustrating the LDM 300. As shown in FIG. 3, the LDM 300 includes a hidden space 301, a conditional module 302, a diffusion module 303, a denoising module 304 and a pixel space 305.

[0047] First, we obtain the hidden variables z0 by mapping the original image x0 to the hidden space 301 by a pre-trained encoder, and then obtain z0 by performing a forward diffusion process by the diffusion module 303. t Get z t , and then provide the new tensor to the denoising module 304 for the reverse denoising process. In the reverse process, the noise-free hidden variable z0 is finally obtained, and then the noise-free image x0 is obtained by mapping it to the pixel space 305 by the pre-trained decoder. The condition module 302 is used to provide the denoising module 304 with conditions of types such as image, text, semantic map, etc.

[0048] The LDM 300 can perform two diffusion processes:

[0049] In the first diffusion process, we first give a noise-free hidden variable z0, and then gradually run the diffusion process. After t steps of diffusion, we give a noise-containing hidden variable z t can be obtained as follows:

[0050] z t =z t-1 +ε=z0+ε0+ε1+···+ε (1) Among them, ε0 is the noise added at the 1st step, ε1 is the noise added at the 2nd step, and ε is the noise added at the tth step. However, this is because Z t This method is inefficient because multiple sampling and calculations are required to obtain the desired value.

[0051] Preferably, the second diffusion process utilizes the derivation of the DDPM reparameterization to directly go from z0 to z by sampling the noise only once, as shown in Equation 2 below: t Get the.

[0052]

number

[0053] Above, the input hidden variable z of the neural network 100 t We have briefly introduced the generation process of L. Below, we will explain the construction of the loss function L of the neural network 100.

[0054] Since the unmasked parts on the mask image block are also significantly different from the original image or line drawing image after rendering, the neural network 100 can reduce the added noise L LDM In addition to being able to accurately predict the noise of image L 画像再構築 Therefore, the loss function L of the neural network 100 consists of two parts:

[0055] L=L LDM +L 画像再構築 (3)

[0056]

number

[0057] The network parameter that needs to be optimized in neural network 100 is θ. As mentioned above, the network parameter θ that needs to be optimized relates to UNET encoders 13, 14 and, optionally, to UNET decoder 21.

[0058] Note that optimizing the network parameter θ involves calculating the gradient ∂L / ∂θ of the network parameter based on the loss function L, and then updating the parameter θ according to the random gradient descent algorithm (θ:θ←θ-μ(∂L / ∂θ)), where μ is a predetermined learning rate.

[0059] By adjusting the network parameters, the neural network 100 is trained until it converges (i.e., the loss function stops dropping) or until a predefined number of training rounds is reached.

[0060] Note that the parameters of UNET encoders 13 and 14 can be obtained directly from pre-trained UNET encoders 11 and 12.

[0061] In addition, the neural network 100 shown in FIG. 1 further includes a Basic Transformer Block (not shown), and the Basic Transformer Block can be considered to be included in the UNET decoder 21.

[0062] A method 400 for training a neural network in an embodiment of the present invention, shown in FIG. 4, will now be described in conjunction with the neural network 100 shown in FIG.

[0063] First, in step 401, a noise image is obtained by adding different noises to the original image. Specifically, in this embodiment, the original image is converted into a noise image using the LDM 300 shown in FIG. 3, and then the hidden variable z t You may also obtain

[0064] Next, in step 402, the noise image is input to a first UNET encoder, the line drawing image corresponding to the original image and the noise image are added by a first adder, and then input to a second UNET encoder, and the noise image and the line drawing image after zero convolution are added by a second adder, and then input to a third UNET encoder.

[0065] Specifically, in this embodiment, the hidden variable z t is input to the UNET encoder 11, and the hidden variables z t are added by adder 22 and then input to UNET encoder 12, and after zero convolution is performed on the line drawing image by zero convolution module 25, the result is added to the hidden variable ZT of the noise image by adder 23, and then input to UNET encoder 13.

[0066] It should be noted that the line drawing image is coded by an encoder (not shown) which participates in training the neural network 100 before being input to the adder 22 and the zero convolution module 25 .

[0067] Next, in step 403, the noise image and a partially masked image corresponding to the original image are added by a third adder, and then input to a fourth UNET encoder. t are added by adder 24 and then input to UNET encoder 14.

[0068] It should be noted that the mask image is encoded by a pre-trained encoder (not shown) before being input to the adder 24 .

[0069] However, depending on the needs, any part of the original image may be masked with a solid color, preferably a black mask.

[0070] Finally, in step 404, the neural network is trained. Specifically, in this embodiment, when the neural network 100 is trained using the loss function L of equations (3) to (5), the parameters of the UNET encoders 13 and 14 and the zero convolution modules 25 and 26 are adjusted, and the parameters of the UNET decoder 21, the text encoder 10, the time encoder 20, and the UNET encoders 11 and 12 are kept unchanged, so that the neural network 100 can converge.

[0071] Optionally, when training the neural network 100, the parameters of the UNET encoders 13, 14, the UNET decoder 21, and the zero convolution modules 25, 26 can be adjusted while keeping the parameters of the text encoder 10, the time encoder 20, and the UNET encoders 11, 12 unchanged, allowing the neural network 100 to converge.

[0072] Also, during training of the neural network 100, the noise image, the mask image and the line-art image correspond to the same original image.

[0073] Furthermore, the output result of the UNET encoder 13 is provided to a second input of the UNET decoder 21 after being zero-convolved by a zero-convolution module 26.

[0074] According to one preferred embodiment, the UNET encoders 11, 12, 13, 14 and the UNET decoder 21 are pre-trained prior to training the neural network 100.

[0075] A method 500 for performing inference using a pre-trained neural network in accordance with an embodiment of the present invention will now be described in conjunction with the neural network 100 shown in FIG.

[0076] First, in step 501, a line drawing image is divided into N image blocks such that adjacent image blocks have overlapping portions, where N is an integer equal to or greater than 2.

[0077] For example, as shown in FIG. 6, a long strip of line art image awaiting input to neural network 100 can be divided into three image blocks with overlapping portions.

[0078] Next, in step 502, a first noise image among the N noise images is input to a first UNET encoder, the first noise image and a first image block of the line drawing image are added by a first adder and then input to a second UNET encoder, the first noise image and the first image block after convolution of the line drawing image are added by a second adder and then input to a third UNET encoder, and the first noise image and the pure color image are added by a third adder and then input to a fourth UNET encoder.

[0079] Specifically, in this embodiment, the hidden variable z t is input to the UNET encoder 11, and the hidden variables z t and the first image block of the line drawing image are added by an adder 22 and then input to the UNET encoder 12. The first image block of the line drawing image is convolved by a zero convolution module 25, and then the hidden variable z t and the hidden variable z t and the pure color image are added by an adder 24 and then input to the UNET encoder 14.

[0080] In this embodiment, the first to Nth noise images are directly sampled using Gaussian noise, and then the corresponding hidden variables z t All we need to do is take as input.

[0081] Also, each image block of the line drawing image is coded by an encoder (not shown) which participates in the training of the neural network 100 before being input to the adder 22 and the zero convolution module 25 .

[0082] Furthermore, the solid color image is coded by a pre-trained coder (not shown) before being input to the adder 24 .

[0083] It should be noted that the zero convolution modules 25, 26 in the neural network 100 shown in FIG. 1 are used only as convolution modules since their parameters are not zero in the inference stage.

[0084] Then, in step 503, a first mask image block is obtained based on a first output image block output from the UNET decoder, which corresponds to the first image block of the line drawing image. Specifically, in this embodiment, the UNET decoder 21 outputs a first output image block corresponding to the first image block of the line drawing image. A first mask image block can be obtained after partially masking the first output image block.

[0085] According to the needs, any part of the output image block of the UNET decoder 21 may be masked with a solid color, preferably with a black mask, for example, if the line drawing image to be divided has a long strip shape in the vertical direction, the lower part of the output image block may be masked, and if the line drawing image to be divided has a long strip shape in the horizontal direction, the right part of the output image block may be masked.

[0086] Also, the colors of the solid color image and the colors for masking the image are the same as the mask colors used when training the neural network 100 .

[0087] Furthermore, the line drawing image block, the mask image block and the solid color image block have the same size.

[0088] Next, in step 504, the second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image by a first adder, and then input to the second UNET encoder, and the results of convolution performed on the second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image by a second adder, and then input to the third UNET encoder.

[0089] Specifically, in this embodiment, the second image block to the Nth image block of the line drawing image are sequentially subjected to the hidden variables z t The second image block to the Nth image block of the line drawing image are convolved by the zero convolution module 23, and the hidden variables z t The sum is added to the signal by an adder 23 and then input to the UNET encoder 13.

[0090] Next, in step 505, the first mask image block to the (N-1)th mask image block are sequentially added to the second noise image to the (N)th noise image by a third adder, and then input into a fourth UNET encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image.

[0091] Specifically, in this embodiment, the first mask image block through the N-1th mask image block are sequentially added to the second noise image through the Nth noise image by the adder 24, and then input to the UNET encoder 14 to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image.

[0092] In addition, the first mask image block to the N-1th mask image block are obtained in the following manner, that is, the parts of the first output image block to the N-1th output image block output from the UNET encoder 21 that overlap with the second image block to the Nth image block of the line drawing image are extracted and used as part of the first to N-1th mask image blocks, and the other parts of the first to N-1th mask image blocks are filled with pure color.

[0093] Finally, in step 506, the N output image blocks output from the UNET decoder are sequentially stitched together to form a complete image. Specifically, in this embodiment, the N output image blocks output from the UNET decoder 21 are stitched together to form a complete image.

[0094] 6 is a diagram showing how mask image blocks are generated from the divided line drawing image blocks, and how a complete image is finally generated after splicing. When performing inference using the neural network 100, the first image block after rendering of the line drawing image output from the UNET decoder 21 is obtained based on the first image block of the line drawing image and the solid color image, and the first mask image block and the second mask image block are obtained by masking the lower parts of the second output image and the third output image output from the UNET decoder 21, respectively. The first output image block to the third output image block are spliced ​​together and then output from the neural network 100.

[0095] Preferably, in this embodiment, the stitching is performed in the manner of weighted fusion, so as to further improve the problem of continuity between output image blocks. Hereinafter, the weighted fusion stitching will be described in detail in conjunction with FIG.

[0096] The principle of weighted fusion stitching is that the closer the overlapping part is to one edge of the output image block, the higher the similarity to the edge, and the farther the overlapping part is from the edge, the higher the similarity to the other edge of the output image block. For example, weighted fusion can be realized by the following formula:

[0097]

number

[0098] The neural network training method 400 in FIG. 4 and the pre-trained neural network inference method 500 in FIG. 5 can greatly alleviate the continuity problem between adjacent image blocks, so that a stitched image with high quality and uniform coloring can be generated, and the content and details of the stitched image will not be changed.

[0099] The above-mentioned methods may be implemented entirely by a computer-executed program, or may be implemented partially or entirely by using hardware and / or firmware. When implemented by using hardware and / or firmware, or when the computer-executable program is installed on a hardware device capable of executing the program, the following described apparatus for training a neural network and for performing inference using a pre-trained neural network are realized. Below, an overview of these apparatus is given without repeating some of the details described above, but it will be understood that these apparatuses are capable of executing the above-mentioned methods, but the methods are not limited to being executed by these apparatuses.

[0100] 8 is a diagram illustrating an apparatus 800 for training a neural network in an embodiment of the present invention. The apparatus 800 includes an acquisition unit 801, an input unit 802, and a training unit 803. The acquisition unit 801 is used to obtain a noise image by adding different noises to an original image. The input unit 802 is used to input the noise image to a first UNET encoder, add a line drawing image corresponding to the original image and the noise image by a first adder before inputting the image to a second UNET encoder, add the noise image and the line drawing image by a second adder before inputting the image to a third UNET encoder, and add the noise image and a partially masked image corresponding to the original image by a third adder before inputting the image to a fourth UNET encoder. The training unit 803 is used to train the neural network in the following manner: adjust parameters of the third UNET encoder and the fourth UNET encoder, and keep the parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, so that the neural network converges; or adjust parameters of the UNET decoder, the third UNET encoder, and the fourth UNET encoder, and keep the parameters of the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, so that the neural network converges.

[0101] The apparatus 800 for training a neural network shown in Fig. 8 corresponds to the method 400 for training a neural network shown in Fig. 4. Therefore, for the relevant details of each unit in the apparatus 800 for training a neural network, reference can be made to the description of the method 400 for training a neural network shown in Fig. 4, and detailed description thereof will be omitted here.

[0102] 9 is a diagram illustrating an apparatus 900 for performing inference using a pre-trained neural network in an embodiment of the present invention. The apparatus 900 includes an image division unit 901, an inference unit 902, and a stitching unit 903. The image division unit 901 is used for dividing a line drawing image into N image blocks, where adjacent image blocks among the N image blocks have overlapping portions, and N is an integer equal to or greater than 2. The inference unit 902 is used to: input a first noise image among the N noise images to a first UNET encoder; add the first noise image and a first image block among the N image blocks of the line drawing image by a first adder and a second adder, respectively, and then input the sum to the second UNET encoder and the third UNET encoder; add the first noise image and a pure color image by a third adder, and then input the sum to the fourth UNET encoder; obtain a first mask image block according to a first output image block output from the UNET decoder, which corresponds to the first image block of the line drawing image; and sequentially add the second image block to the Nth image block of the line drawing image to the second noise image to the Nth noise image by the first adder and the second adder, respectively. The first mask image block to the N-1th mask image block are sequentially added to the second noise image to the Nth noise image by a third adder, and then input to the second UNET encoder and the third UNET encoder, respectively, to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image, among which, the first output image block to the N-1th output image block output from the UNET encoder are respectively extracted and overlapped with the second image block to the Nth image block of the line drawing image as a part of the first to the N-1th mask image block, and the other parts of the first to the N-1th mask image blocks are filled with a solid color to obtain the first mask image block to the N-1th mask image block. The splicing unit 903 is used to sequentially splice the N output image blocks output from the UNET decoder to form a complete image.

[0103] The apparatus 900 for making inference using a pre-trained neural network shown in Fig. 9 corresponds to the method 500 for making inference using a pre-trained neural network shown in Fig. 5. Therefore, for relevant details of each unit in the apparatus 900 for making inference using a pre-trained neural network, reference can be made to the description of the method 500 for making inference using a pre-trained neural network shown in Fig. 5, and detailed description thereof will be omitted here.

[0104] Furthermore, each unit in the above-mentioned device may be configured by software, firmware, hardware, or a combination thereof. Specific means and methods that can be used during configuration are well known to those skilled in the art, so detailed explanations are omitted here. When realized by software or firmware, the programs that constitute the software are installed from a storage medium or network into a computer having a dedicated hardware structure (e.g., the general-purpose computer 200 shown in FIG. 10), and the computer can execute various functions when various programs are installed.

[0105] Also, as will be apparent, the process of each operation of the method according to the present invention can be realized in the form of a computer executable program stored in various machine-readable storage media.

[0106] Furthermore, the object of the present invention may be realized in the following manner: a storage medium storing executable program code is provided directly or indirectly to a system or device, and the program code is read and executed by a computer or central processing unit (CPU) in the system or device. In this case, as long as the system or device has a function capable of executing a program, the embodiment of the present invention is not limited to a program, and the program may be in any form, such as an object-oriented program, a program executed by an interpreter, a script program provided to an OS (operating system), etc.

[0107] Such machine-readable storage media include, but are not limited to, various types of memory devices and storage units, semiconductor devices, magnetic disk drives such as magnetic and magneto-optical disks, and other media suitable for storing information.

[0108] In addition, the computer can also connect to a corresponding website on the Internet, download and install the computer program code according to the present invention on the computer, and then run the program to realize the technical solution of the present invention.

[0109] FIG. 10 is a diagram showing an exemplary configuration of a general-purpose personal computer capable of implementing the method and / or apparatus according to an embodiment of the present invention.

[0110] The computer 200 may be, for example, a computer system. Note that the computer 200 is merely an example and does not limit the scope or functionality of the method and apparatus according to the present invention. Furthermore, the computer 200 does not depend on any module, assembly, etc., or combination thereof in the above-mentioned method and apparatus.

[0111] 10, a central processing unit (CPU) 201 performs various processes based on a program stored in a ROM 202 or a program loaded from a storage unit 208 to a RAM 203. The RAM 203 can also store data required when the CPU 201 performs various processes, depending on the needs. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output interface 205 is also connected to the bus 204.

[0112] Further, the following components are connected to the input / output interface 205: an input unit 206 including a keyboard, an output unit 207 including a display such as a liquid crystal display (LCD) and a speaker, a storage unit 208 including a hard disk, and a communication unit 209 including a network interface card such as a LAN card and a modem. The communication unit 209 performs communication processing via a network such as the Internet or a LAN. A drive 210 may be connected to the input / output interface 205 according to needs. A removable medium 211, such as a semiconductor memory, can be set in the drive 210 as needed, and a computer program read from the removable medium 211 can be installed in the storage unit 208.

[0113] The present invention further provides a program product including a machine-readable instruction code. When such instruction code is read and executed by a machine, it can execute the method in the above-mentioned embodiment of the present invention. Accordingly, various storage media that carry such a program product, such as magnetic disks (including floppy disks (registered trademark)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs (registered trademarks)), and semiconductor storage devices, are also included in the present invention.

[0114] The above-mentioned storage medium may include, for example, a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory device, and the like, but is not limited to these.

[0115] Additionally, each operation (process) in the above-described method may be implemented in the form of a computer-executable program stored in various machine-readable storage media.

[0116] Furthermore, with respect to the above-mentioned embodiments, the following supplementary notes are disclosed.

[0117] (Appendix 1) 1. An apparatus for training a neural network, comprising: The neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and the output end of the second UNET encoder, the output end of the third UNET encoder and the output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, such an apparatus comprises: an obtaining unit configured to obtain a noise image by adding different noises to the original image; An input unit, Input the noise image to a first UNET encoder; A line drawing image corresponding to an original image and a noise image are added by a first adder, and then input to a second UNET encoder; The noise image and the line drawing image are added by a second adder, and then input to a third UNET encoder; and a noise image and a partially masked image corresponding to the original image are added by a third adder before being input to a fourth UNET encoder; and Including training units, The training unit is configured to train the neural network in the following manner: Adjusting parameters of the third UNET encoder and the fourth UNET encoder, and keeping parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, thereby allowing the neural network to converge; or The apparatus adjusts parameters of the UNET decoder, the third UNET encoder, and the fourth UNET encoder, and keeps parameters of the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, thereby causing the neural network to converge.

[0118] (Appendix 2) 2. The apparatus of claim 1, the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder; Among them, the presentation is input to the text encoder and the time step is input to the time encoder to obtain tensors of the presentation and the time step, respectively; Wherein, each tensor of the presentation and time steps is provided to a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, and a UNET decoder, respectively.

[0119] (Appendix 3) 3. The apparatus of claim 2, The neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third UNET encoder and a second input of the UNET decoder; Wherein, the training unit is further configured to adjust parameters of the first zero convolution module and parameters of the second zero convolution module during training; Wherein, the line drawing image is provided to the second adder after being zero-convolved by the first zero-convolution module.

[0120] (Appendix 4) 4. The apparatus of claim 3, The apparatus, wherein a total loss function of the neural network is constructed as a sum of a first loss function and a second loss function, in which the first loss function is a sum-of-squares loss function between a randomly sampled noise and a noise estimated for the noisy image, and the second loss function is a sum-of-squares loss function between a hidden variable corresponding to an original image without noise and a hidden variable of a reconstructed image after noise removal.

[0121] (Appendix 5) 5. An apparatus according to any one of claims 1 to 4, comprising: The apparatus obtains hidden variables of the original image by another pre-trained encoder before the noise image is input to the first UNET encoder, the first adder, the second adder and the third adder, and then adds noise to the hidden variables to obtain hidden variables corresponding to the noise image.

[0122] (Appendix 6) 4. The apparatus of claim 3, The line drawing image is encoded by another encoder that participates in training the neural network before being input to the first adder and the first zero convolution module.

[0123] (Appendix 7) 5. An apparatus according to any one of claims 1 to 4, comprising: The masked image is then coded by another coder that participates in training the neural network before being input to a third adder.

[0124] (Appendix 8) 6. An apparatus according to any one of claims 1 to 5, comprising: The first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder are pre-trained prior to training the neural network.

[0125] (Appendix 9) 1. An apparatus for performing inference using a pre-trained neural network, comprising: The neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and the output end of the second UNET encoder, the output end of the third UNET encoder and the output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, such an apparatus comprises: an image segmentation unit configured to segment the line drawing image into N image blocks, where adjacent image blocks among the N image blocks have overlapping portions with each other, where N is an integer greater than or equal to 2; and Including an inference unit, The inference unit is configured to perform the following operations: inputting a first noise image of the N noise images to a first UNET encoder; A first image block among the N image blocks of the first noise image and the line drawing image is added by a first adder and a second adder, respectively, and then input to a second UNET encoder and a third UNET encoder, respectively; The first noise image and the pure color image are added by a third adder, and then input to a fourth UNET encoder; Obtain a first mask image block based on a first output image block output from the UNET decoder, the first mask image block corresponding to a first image block of the line drawing image; The second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image of the N noise images by the first adder and the second adder, respectively, and then input to the second UNET encoder and the third UNET encoder, respectively; the first mask image block through the (N-1)th mask image block are sequentially added to the second noise image through the (N)th noise image by a third adder, and then input to a fourth UNET encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image; The device extracts overlapping parts of the first output image block to the N-1th output image block output from the UNET encoder and the second image block to the Nth image block of the line drawing image, respectively, to form part of the first to N-1th mask image blocks, and fills other parts of the first to N-1th mask image blocks with a solid color to obtain the first mask image block to the N-1th mask image block.

[0126] (Appendix 10) 10. The apparatus of claim 9, further comprising: The method further includes a splicing unit, the splicing unit being configured to sequentially splice the N output image blocks output from the UNET decoder to form a complete image; Wherein, the stitching unit is further configured to stitch the N output image blocks in a weighted fusion manner, so that parts of the N output image blocks closer to an edge have higher weights.

[0127] (Appendix 11) 10. The apparatus of claim 9, further comprising: the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder; Wherein, input the presentation to the text encoder and input the time step to the time encoder to obtain the presentation and time step tensors, respectively; Wherein, each tensor of the presentation and time steps is provided to a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, and a UNET decoder, respectively.

[0128] (Appendix 12) 10. The apparatus of claim 9, further comprising: The neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between an output of the third UNET encoder and a second input of the UNET decoder; Wherein, the first zero convolution module and the second zero convolution module perform non-zero parameter convolution during the inference period of the neural network; Wherein, the N image blocks of the line drawing image are convolved by a first zero convolution module and then provided to a second adder, the device.

[0129] (Appendix 13) 1. A method for training a neural network, comprising: The neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and the output end of the second UNET encoder, the output end of the third UNET encoder and the output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, and the method includes: Obtain a noisy image by adding different noises to the original image; Input the noise image to the first UNET encoder; The line image and the noise image corresponding to the original image are added by a first adder, and then input to a second UNET encoder; The noise image and the line drawing image are added by a second adder, and then input to a third UNET encoder; The noise image and the partially masked image corresponding to the original image are added by a third adder, and then input to a fourth UNET encoder; and training a neural network in the following manner: That is, by adjusting the parameters of the third UNET encoder and the fourth UNET encoder, and keeping the parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, the neural network is allowed to converge; or A method for allowing a neural network to converge by adjusting parameters of a UNET decoder, a third UNET encoder, and a fourth UNET encoder, and keeping parameters of a text encoder, a time encoder, a first UNET encoder, and a second UNET encoder unchanged.

[0130] (Appendix 14) 14. The method of claim 13, comprising: the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder, and the UNET decoder; wherein, the neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third UNET encoder and the second input end of the UNET decoder, wherein, training the neural network further includes adjusting parameters of the first zero convolution module and parameters of the second zero convolution module.

[0131] (Appendix 15) 15. The method according to claim 14, The method, wherein a total loss function of the neural network is constructed as a sum of a first loss function and a second loss function, in which the first loss function is a sum-of-squares loss function between a randomly sampled noise and a noise estimated for the noisy image, and the second loss function is a sum-of-squares loss function between a hidden variable corresponding to an original image without noise and a hidden variable of a reconstructed image after noise removal.

[0132] (Appendix 16) 1. A method for performing inference using a pre-trained neural network, comprising: The neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and the output end of the second UNET encoder, the output end of the third UNET encoder and the output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, and the method includes: Dividing the line drawing image into N image blocks, and adjacent image blocks among the N image blocks have overlapping portions, where N is an integer greater than or equal to 2; input a first noise image of the N noise images to a first UNET encoder; A first image block among the N image blocks of the first noise image and the line drawing image is added by a first adder and a second adder, respectively, and then inputted into a second UNET encoder and a third UNET encoder, respectively; The first noise image and the pure color image are added by a third adder, and then input to a fourth UNET encoder; Obtain a first mask image block according to a first output image block corresponding to a first image block of the line drawing image output from the UNET decoder; The second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image of the N noise images by the first adder and the second adder, respectively, and then input to the second UNET encoder and the third UNET encoder, respectively; and and sequentially adding the first mask image block to the (N-1)th mask image block to the second noise image to the (N-1)th noise image by a third adder, and then inputting the resultant to a fourth UNET encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image; The method includes extracting overlapping parts of the first output image block to the N-1th output image block output from the UNET encoder and the second image block to the Nth image block of the line drawing image, respectively, to obtain the first mask image block to the N-1th mask image block as part of the first mask image block to the N-1th mask image block, and filling other parts of the first mask image block to the N-1th mask image block with a solid color.

[0133] (Appendix 17) 17. The method of claim 16, further comprising: A method comprising sequentially stitching together N output image blocks output from a UNET decoder to form a complete image in a weighted fusion manner, such that portions of the N output image blocks closer to edges have higher weights.

[0134] (Appendix 18) 18. The method according to claim 16 or 17, The text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder, where the input of the input of the text encoder and the input of the time step are input of the time step to obtain the tensors of the input of the time step and the tensors of the time step, which are respectively provided to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder; wherein the neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to a second adder, and the second zero convolution module is connected between an output end of a third UNET encoder and a second input end of a UNET decoder, wherein the first zero convolution module and the second zero convolution module perform non-zero parameter convolution during an inference period of the neural network.

[0135] (Appendix 19) A computer-readable storage medium having a program stored therein, A computer-readable storage medium, the program, when executed by a processor, causing a computer to implement a method according to any one of claims 13 to 15.

[0136] (Appendix 20) A computer-readable storage medium having a program stored therein, A computer-readable storage medium, the program, when executed by a processor, causing a computer to implement a method according to any one of claims 16 to 18.

[0137] Although the preferred embodiment of the present invention has been described above, the present invention is not limited to this embodiment, and any modification to the present invention falls within the technical scope of the present invention as long as it does not depart from the spirit of the present invention.

Claims

1. 1. An apparatus for training a neural network, comprising: the neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and an output end of the second UNET encoder, an output end of the third UNET encoder and an output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, the device an obtaining unit configured to obtain a noise image by adding different noises to the original image; An input unit, inputting the noise image to the first UNET encoder; a line drawing image corresponding to the original image and the noise image are added by the first adder, and the result is input to the second UNET encoder; The noise image and the line drawing image are added by the second adder, and then input to the third UNET encoder; and an input unit configured to add the noise image and a partially masked image corresponding to the original image by the third adder before inputting the noise image and the partially masked image corresponding to the original image to the fourth UNET encoder; and A training unit comprising: adjusting parameters of the third UNET encoder and the fourth UNET encoder, and keeping parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder and the second UNET encoder unchanged, thereby allowing the neural network to converge; or Adjusting parameters of the UNET decoder, the third UNET encoder, and the fourth UNET encoder, and keeping parameters of the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, thereby allowing the neural network to converge. a training unit configured to train the neural network by:

2. 2. The apparatus of claim 1, the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder; inputting a prompt to the text encoder and a time step to the temporal encoder to obtain tensors for the prompt and the time step, respectively; The apparatus, wherein the tensors for each of the representations and the time steps are respectively provided to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder.

3. 3. The apparatus of claim 2, The neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third UNET encoder and a second input of the UNET decoder; The line drawing image is provided to the second adder after being zero-convolved by the first zero-convolution module.

4. 4. An apparatus according to any one of claims 1 to 3, comprising: an overall loss function of the neural network is constructed as a sum of a first loss function and a second loss function, the first loss function being a sum-of-squares loss function between randomly sampled noise and a noise estimated for the noisy image, and the second loss function being a sum-of-squares loss function between hidden variables corresponding to the original image without noise and hidden variables of a reconstructed image after noise has been removed.

5. 1. An apparatus for performing inference using a pre-trained neural network, comprising: the neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and an output end of the second UNET encoder, an output end of the third UNET encoder and an output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, the device an image segmentation unit configured to segment the line drawing image into N image blocks, where adjacent image blocks among the N image blocks have overlapping portions, and N is an integer greater than or equal to 2; and Including an inference unit, The inference unit comprises: inputting a first noise image of the N noise images to the first UNET encoder; The first noise image and a first image block among the N image blocks of the line drawing image are added by the first adder and the second adder, respectively, and then input to the second UNET encoder and the third UNET encoder, respectively; The first noise image and the pure color image are added by the third adder, and then input to the fourth UNET encoder; Obtain a first mask image block based on a first output image block output from the UNET decoder and corresponding to a first image block of the line drawing image; the second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image of the N noise images by the first adder and the second adder, respectively, and then inputted to the second UNET encoder and the third UNET encoder, respectively; The first mask image block to the (N-1)th mask image block are sequentially added to the second noise image to the (N)th noise image by the third adder, and then input to the fourth UNET encoder, thereby sequentially obtaining N output image blocks corresponding to the N image blocks of the line drawing image. The method is configured to: An apparatus for obtaining the first mask image block to the N-1th mask image block by extracting portions of the first output image block to the N-1th output image block output from a UNET encoder that overlap with the second image block to the Nth image block of the line drawing image, respectively, and forming the portions into first to N-1th mask image blocks, and filling other portions of the first to N-1th mask image blocks with a solid color.

6. 6. The apparatus of claim 5, The apparatus further includes a stitching unit configured to stitch together the N output image blocks output from the UNET decoder in sequence to form a complete image.

7. 6. The apparatus of claim 5, the text encoder and the time encoder are connected to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder; inputting an excerpt to the text encoder and a time step to the temporal encoder to obtain tensors for the excerpt and the time step, respectively; The apparatus, wherein the tensors for each of the representations and the time steps are respectively provided to the first UNET encoder, the second UNET encoder, the third UNET encoder, the fourth UNET encoder and the UNET decoder.

8. 8. An apparatus according to any one of claims 5 to 7, comprising: The neural network further includes a first zero convolution module and a second zero convolution module, the first zero convolution module being connected to the second adder, and the second zero convolution module being connected between an output of the third UNET encoder and a second input of the UNET decoder; the first zero convolution module and the second zero convolution module perform non-zero parameter convolution during inference of the neural network; The N image blocks of the line drawing image are provided to the second adder after being convolved by the first zero convolution module.

9. A program, Causing a computer to execute a method for training a neural network; the neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and an output end of the second UNET encoder, an output end of the third UNET encoder and an output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, the method comprising: Obtain a noisy image by adding different noises to the original image; inputting the noise image to the first UNET encoder; A line drawing image corresponding to the original image and the noise image are added by the first adder, and then input to the second UNET encoder; The noise image and the line drawing image are added by the second adder, and then input to the third UNET encoder; The noise image and a partially masked image corresponding to the original image are added by the third adder, and then input to the fourth UNET encoder; and training the neural network; Training the neural network includes: adjusting parameters of the third UNET encoder and the fourth UNET encoder, and keeping parameters of the UNET decoder, the text encoder, the time encoder, the first UNET encoder and the second UNET encoder unchanged, thereby allowing the neural network to converge; or Adjusting parameters of the UNET decoder, the third UNET encoder, and the fourth UNET encoder, and keeping parameters of the text encoder, the time encoder, the first UNET encoder, and the second UNET encoder unchanged, thereby allowing the neural network to converge. This is achieved by the program.

10. A program, Causing a computer to perform a method for performing inference using a pre-trained neural network; the neural network includes a text encoder, a time encoder, a first UNET encoder, a second UNET encoder, a third UNET encoder, a fourth UNET encoder, a UNET decoder, a first adder, a second adder and a third adder, an output end of the first UNET encoder is connected to a first input end of the UNET decoder, the first adder is connected to an input end of the second UNET encoder, the second adder is connected to an input end of the third UNET encoder, the third adder is connected to an input end of the fourth UNET encoder, and an output end of the second UNET encoder, an output end of the third UNET encoder and an output end of the fourth UNET encoder are connected to a second input end of the UNET decoder, the method comprising: Dividing a line drawing image into N image blocks, adjacent image blocks among the N image blocks have overlapping portions, N being an integer equal to or greater than 2; inputting a first noise image of the N noise images to the first UNET encoder; The first noise image and a first image block among the N image blocks of the line drawing image are respectively added by the first adder and the second adder, and then inputted to the second UNET encoder and the third UNET encoder, respectively; The first noise image and the pure color image are added by the third adder, and then input to the fourth UNET encoder; Obtaining a first mask image block based on a first output image block output from the UNET decoder and corresponding to a first image block of the line drawing image; The second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image of the N noise images by the first adder and the second adder, respectively, and then input to the second UNET encoder and the third UNET encoder, respectively; and and sequentially adding the first mask image block to the (N-1)th mask image block to the second noise image to the (N)th noise image by the third adder, and then inputting the resultant data to the fourth UNET encoder, thereby sequentially obtaining N output image blocks corresponding to the N image blocks of the line drawing image; A program for obtaining the first mask image block to the N-1th mask image block by extracting portions of the first output image block to the N-1th output image block output from a UNET encoder that overlap with the second image block to the Nth image block of the line drawing image, respectively, and making them into portions of the first to N-1th mask image blocks, and filling other portions of the first to N-1th mask image blocks with a solid color.

Images