Neural network training device, inference device and computer readable storage medium

Abstract

The invention discloses a neural network training device, an inference device and a medium. The neural network comprises a text encoder, a time encoder, first to fourth Unet encoders, a Unet decoder, and first to third adders. The training apparatus includes: an obtaining unit that obtains a noise image by adding different noise to an original image; an input unit which inputs the noise image into the first Unet encoder, adds the line draft image corresponding to the original image and the noise image through the first and second adders respectively, and then inputs the line draft image and the noise image into the second and third Unet encoders respectively; adding the noise image and the partially masked image corresponding to the original image through a third adder, and inputting the added image into a fourth Unet encoder; and a training unit that adjusts parameters of the third and fourth Unet encoders, and maintains parameters of the text encoder, the time encoder, the first and second Unet encoders, respectively, such that the neural network converges, in which the parameters of the Unet decoders can be adjusted or maintained unchanged during training.

Description

Technical Field

[0001] The present disclosure relates to the field of neural networks, and in particular to a technique for rendering line drawings using neural networks. Background Art

[0002] Coloring line drawings is an essential part of the animation creation pipeline. However, it is difficult to ensure both quality and continuity when rendering long line drawings in horizontal or vertical directions.

[0003] Currently, two methods have been proposed to improve the continuity between adjacent image blocks while maintaining the quality of the rendered image.

[0004] The first method involves splitting the long line drawing into multiple blocks, rendering each block separately, and then stitching the rendered blocks together. Although this solves the problem of requiring large GPU memory, the continuity of the stitched image is poor because there are obvious seams between adjacent blocks.

[0005] The second method involves rendering a reduced version of the original long line drawing, and then using the ControlNet Tile model to improve the details of the rendered image of the reduced version of the line drawing. This improves the continuity of the image, but changes the content and details in the line drawing. Summary of the invention

[0006] A brief overview of the disclosure is provided below in order to provide a basic understanding of certain aspects of the disclosure. It should be understood that this overview is not an exhaustive overview of the disclosure. It is not intended to identify key or important parts of the disclosure, nor is it intended to limit the scope of the disclosure. Its purpose is simply to provide certain concepts in a simplified form as a prelude to a more detailed description discussed later.

[0007] According to one aspect of the present disclosure, a device for training a neural network is provided. The neural network includes a text encoder, a time encoder, a first U-shaped network (Unet) encoder, a second Unet encoder, a third Unet encoder, a fourth Unet encoder, a Unet decoder, a first adder, a second adder, and a third adder, the output of the first Unet encoder is connected to the first input of the Unet decoder, the first adder is connected to the input of the second Unet encoder, the second adder is connected to the input of the third Unet encoder, the third adder is connected to the input of the fourth Unet encoder, and the output of the second Unet encoder, the output of the third Unet encoder, and the output of the fourth Unet encoder are connected to the second input of the Unet decoder. The apparatus includes: an obtaining unit configured to obtain a noise image by adding different noises to an original image; an input unit configured to: input the noise image into a first Unet encoder, input the line draft image corresponding to the original image and the noise image into a second Unet encoder after adding them through a first adder, input the noise image and the line draft image into a third Unet encoder after adding them through a second adder, and input the noise image and the partially masked image corresponding to the original image into a fourth Unet encoder after adding them through a third adder; and a training unit configured to train a neural network by: adjusting parameters of the third Unet encoder and the fourth Unet encoder, and keeping the respective parameters of the Unet decoder, the text encoder, the time encoder, the first Unet encoder and the second Unet encoder unchanged, so that the neural network converges, or adjusting the respective parameters of the Unet decoder, the third Unet encoder and the fourth Unet encoder, and keeping the respective parameters of the text encoder, the time encoder, the first Unet encoder and the second Unet encoder unchanged, so that the neural network converges.

[0008] Preferably, the text encoder and the time encoder are connected to the first Unet encoder, the second Unet encoder, the third Unet encoder, the fourth Unet encoder and the Unet decoder. The text encoder is input with prompts, and the time encoder is input with time steps to obtain respective tensors of prompts and time steps. The respective tensors of prompts and time steps are provided to the first Unet encoder, the second Unet encoder, the third Unet encoder, the fourth Unet encoder and the Unet decoder respectively.

[0009] Preferably, the neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third Unet encoder and the second input end of the Unet decoder. The line drawing image is provided to the second adder after zero convolution through the first zero convolution module.

[0010] Preferably, the training unit is further configured to adjust parameters of the first zero convolution module and parameters of the second zero convolution module during training.

[0011] Preferably, the partially masked image is obtained by randomly masking a portion of the original image with a solid color.

[0012] Preferably, before the noise image is input into the first Unet encoder, the first adder, the second adder and the third adder, the latent variables of the original image are obtained by another pre-trained encoder, and then noise is added to the latent variables to obtain the latent variables corresponding to the noise image.

[0013] Preferably, the line drawing image is encoded by a further encoder participating in the training of the neural network before being input into the first adder and the first zero convolution module.

[0014] Preferably, the partially masked image is encoded by a further encoder participating in the training of the neural network before being input to the third adder.

[0015] Preferably, the first Unet encoder, the second Unet encoder, the third Unet encoder, the fourth Unet encoder and the Unet decoder are pre-trained before training the neural network.

[0016] Preferably, the total loss function of the neural network is constructed as the sum of a first loss function and a second loss function, wherein the first loss function is a square sum loss function between randomly sampled noise and noise estimated for a noisy image, and the second loss function is a square sum loss function between latent variables corresponding to an original image without noise and latent variables of a reconstructed image after noise removal.

[0017] According to another aspect of the present disclosure, there is provided a device for performing reasoning using the above-mentioned pre-trained neural network. The device includes: an image segmentation unit, which is configured to: segment the line draft image into N image blocks, where adjacent image blocks in the N image blocks have overlapping parts, where N is an integer greater than or equal to 2; and an inference unit, which is configured to perform the following operations: input the first noise image in the N noise images into the first Unet encoder, add the first noise image and the first image block in the N image blocks of the line draft image through the first adder and the second adder respectively, and then input them into the second Unet encoder and the third Unet encoder respectively, add the first noise image and the pure color image through the third adder, and then input them into the fourth Unet encoder, obtain a first mask image block based on the first output image block corresponding to the first image block of the line draft image output from the Unet decoder, and sequentially add the second image block to the Nth image block of the line draft image. The first mask image block to the N-1th mask image block are sequentially added with the second noise image to the N-th noise image in the N noise images through the first adder and the second adder, and then input into the second Unet encoder and the third Unet encoder respectively; the first mask image block to the N-1th mask image block are sequentially added with the second noise image to the N-th noise image through the third adder, and then input into the fourth Unet encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line draft image, wherein the first output image block to the N-1th output image block outputted from the Unet encoder are respectively extracted by overlapping parts with the second image block to the N-th image block of the line draft image as a part of the first to the N-1th mask image blocks, and the other part of the first to the N-1th mask image blocks is filled with a solid color, so as to obtain the first mask image block to the N-1th mask image block.

[0018] Preferably, the apparatus for inference further comprises a splicing unit configured to sequentially splice the N output image blocks output from the Unet decoder into a complete image.

[0019] Preferably, the splicing unit is further configured to splice the N output image blocks in a weighted fusion manner, so that parts of the N output image blocks closer to the edge have higher weights.

[0020] Preferably, before any image used to obtain N noise images is input into the first Unet encoder, the first adder, the second adder and the third adder, a latent variable of the arbitrary image is obtained by another pre-trained encoder, and then different noises, such as Gaussian noise, are added to the latent variables to obtain N noise images.

[0021] Preferably, the N image blocks of the line drawing image are encoded by another encoder participating in the training of the neural network before being input into the first adder and the first zero convolution module, wherein the parameters of the first zero convolution module and the second zero convolution module are not zero during the inference of the neural network.

[0022] Preferably, before being input into the third adder, the N-1 mask image blocks are encoded by another encoder participating in the training of the neural network to obtain latent variables of the N-1 mask image blocks.

[0023] Preferably, the colors of the solid color image and the colors of the first to N-1th mask image blocks are the same as the colors used for the mask image when training the neural network.

[0024] Preferably, the N image blocks of the line drawing image, the N output image blocks and the solid color image are of the same size.

[0025] According to other aspects of the present disclosure, corresponding computer program codes, computer-readable storage media and computer program products are also provided.

[0026] Through the device for training a neural network and the device for performing reasoning using a pre-trained neural network of the present disclosure, it is possible to improve the continuity of a rendered image, thereby improving the quality of the rendered image while maintaining the content and details in the line drawing.

[0027] These and other advantages of the present disclosure will become more apparent through the following detailed description of preferred embodiments of the present disclosure in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to further illustrate the above and other advantages and features of the present disclosure, the specific embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings. The drawings together with the following detailed description are included in this specification and form a part of this specification. Elements with the same function and structure are represented by the same reference numerals. It should be understood that these drawings only describe typical examples of the present disclosure and should not be regarded as limiting the scope of the present disclosure. In the drawings:

[0029] Figure 1 schematically illustrates a neural network according to an embodiment of the present disclosure;

[0030] Figure 2 The forward process and the reverse process of the Denoising Diffusion Probability Model (DDPM) are schematically shown;

[0031] Figure 3The Latent Diffusion Model (LDM) is schematically shown;

[0032] Figure 4 A flowchart of a method for training a neural network according to an embodiment of the present disclosure is shown;

[0033] Figure 5 A flowchart of a method for performing reasoning using a pre-trained neural network according to an embodiment of the present disclosure is shown;

[0034] Figure 6 Schematically shows Figure 5 Image segmentation and masking in the method;

[0035] Figure 7 Schematically shows Figure 5 Weighted fusion of image stitching in the method;

[0036] Figure 8 is a block diagram of an apparatus for training a neural network according to an embodiment of the present disclosure;

[0037] Fig. 9 is a block diagram of an apparatus for performing inference using a pre-trained neural network according to an embodiment of the present disclosure;

[0038] Fig.10 is a block diagram of an exemplary structure of a general-purpose personal computer in which methods and / or devices according to embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0039] Exemplary embodiments of the present disclosure will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual implementation are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the process of developing any such actual implementation in order to achieve the developer's specific goals, such as meeting those constraints related to the system and business, and these constraints may vary from implementation to implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the content of this disclosure.

[0040] It is also necessary to explain here that, in order to avoid obscuring the present disclosure due to unnecessary details, only the device structure and / or processing steps closely related to the scheme according to the present disclosure are shown in the accompanying drawings, while other details that are not very relevant to the present disclosure are omitted.

[0041] In order to solve the problems of the prior art, the present disclosure proposes a novel method for improving continuity. First, ControlNet is extended to support multiple controls including line control and rendering control, where the continuity between adjacent image blocks is improved by giving previously rendered image blocks as hints to improve rendering control and improve image quality. In addition, the continuity between image blocks is further improved by weighted fusion.

[0042] The neural network according to the present disclosure is based on a pre-trained model, such as ControlNet v1.1 AnimeLineart, to utilize the image knowledge of the pre-trained model; use different neural network branches to control line drafts and rendering respectively; and freeze the weights of the pre-trained model to avoid catastrophic forgetting problems.

[0043] Figure 1 A neural network 100 according to an embodiment of the present disclosure is schematically shown. Figure 1 As shown, the neural network 100 includes Unet encoders 11, 12, 13, 14; a text encoder 10; a temporal encoder 20; a Unet decoder 21; and adders 22, 23, 24. The output of the Unet encoder 11 is connected to a first input of the Unet decoder 21. The output of the adder 22 is connected to the input of the Unet encoder 12, the output of the adder 23 is connected to the input of the Unet encoder 13, and the output of the adder 24 is connected to the input of the Unet encoder 14. The outputs of the Unet encoders 12, 13, 14 are connected to a second input of the Unet decoder 21. The output of the Unet decoder 21 provides an inference result of the neural network 100.

[0044] It should be understood that the text encoder 10 and the temporal encoder 20 are connected to each of the Unet encoders 11 , 12 , 13 , 14 and the Unet decoder 21 .

[0045] As is known, the hint is input to the text encoder 10 to obtain a hint tensor, and the time step is input to the temporal encoder 20 to obtain a time step tensor. The hint and time step respective tensors are provided to each of the Unet encoders 11, 12, 13, 14 and the Unet decoder 21, respectively. It should be understood that the content of the hint may include various features of the image to be rendered.

[0046] It should also be understood that although Figure 1 A text encoder 10 and a time encoder 20 are shown in the figure, but the present disclosure is not limited thereto, and multiple text encoders 10 and multiple time encoders 20 may also be provided to be used for each of the Unet encoders 11, 12, 13, 14 and the Unet decoder 21, respectively.

[0047] According to a preferred embodiment, the neural network 100 further comprises zero convolution modules 25, 26. The output of the zero convolution module 25 is connected to the input of the adder 23, and the zero convolution module 26 is connected between the output of the Unet encoder 13 and the second input of the Unet decoder 21.

[0048] During the training of the neural network 100, noise is first added to the original image to obtain a noisy image, and then the latent variable z of the noisy image is obtained. t Hidden variable z t is input to the Unet encoder 11 and adders 22, 23, 24. The line drawing image corresponding to the original image is input to the adder 22, and is input to the adder 23 after being zero-convolved by the zero-convolution module 25. Hidden variable z t The line drawing image is added to the adder 21 and then input into the Unet encoder 12. Hidden variable z t The zero-convolved line drawing image is added to the adder 23 and then input into the Unet encoder 13. Hidden variable z t The partially masked original image is added to the Unet encoder 14 through the adder 24. The output of the Unet encoder 11 is provided to the first input of the Unet decoder 21, and the outputs of the Unet encoders 12 and 14 are provided to the second input of the Unet decoder 21, and the output of the Unet encoder 13 is provided to the second input of the Unet decoder 21 after zero convolution through the zero convolution module 26.

[0049] During training, only the parameters of the Unet encoders 13, 14 and the zero convolution modules 25, 26 can be adjusted, while keeping the parameters of the Unet decoder 21, text encoder 10, time encoder 20, and Unet encoders 11, 12 unchanged, so that the neural network 100 converges.

[0050] Optionally, during training, the respective parameters of the Unet encoders 13, 14, zero convolution modules 25, 26 and Unet decoder 21 can be adjusted while keeping the respective parameters of the text encoder 10, temporal encoder 20, Unet encoders 11, 12 unchanged so that the neural network 100 converges.

[0051] It should be understood that the above parameters include weights and biases.

[0052] It should be noted that during training, the mask image may be obtained by masking a portion of the original image, such as any one of the upper portion, the lower portion, the left portion, and the right portion, with a solid color, preferably black.

[0053] It should be understood that the zero convolution modules 25, 26 mean that the weights of the modules are 0 only in the first iteration of training the neural network 100, and are not zero during subsequent iterations and during inference.

[0054] It should also be understood that the line draft image is encoded by an encoder (not shown) participating in the neural network training before being input to the adder 22 and the zero convolution module 25, and the mask image is characterized by a pre-trained encoder (not shown) before being input to the adder 24. The above encoding of the line draft image and the mask image can use any suitable existing technology and will not be repeated here.

[0055] It should also be understood that the original image can be noised by using the existing DDPM to obtain a noisy image, and the original image can also be converted into a latent variable z by using the pre-trained autoencoder in the LDM t And then the latent variable is added with noise by DDPM. In order to facilitate understanding of the present disclosure, the process of adding noise to the original image and obtaining the latent variable of the image after adding noise is briefly introduced below.

[0056] Figure 2 The forward process and reverse process of DDPM are schematically shown. Figure 2 As shown, in the forward process from right to left, from X0 to X T Noise is continuously added to the image for diffusion, thus obtaining a noisy image. In the reverse process from left to right, from X T To X0, the noise is gradually removed from the noisy image to obtain a noise-free image. The forward process and the reverse process can occur in both the pixel space and the latent space.

[0057] According to an embodiment of the present disclosure, a DDPM-based LDM is used to perform forward diffusion and reverse denoising processes in a latent space.

[0058] Figure 3 Schematically shows a block diagram of LDM 300. Figure 3 As shown, LDM 300 includes latent space 301, condition module 302, diffusion module 303, denoising module 304 and pixel space 305.

[0059] First, the original image x0 is mapped to the latent space 301 through the pre-trained encoder to obtain the latent variable z0, and then the forward diffusion process is performed through the diffusion module 303 to obtain z T , z TAdd or concatenate to obtain a new tensor, and provide the new tensor to the denoising module 304 for the reverse denoising process. In the reverse process, after finally obtaining the noise-free latent variable z0, it is mapped back to the pixel space 305 through the pre-trained decoder to obtain the noise-free image x0. The condition module 302 is used to provide the denoising module 304 with conditions of the type of image, text, and semantic map.

[0060] The following two diffusion processes may be implemented in LDM 300 .

[0061] In the first diffusion process, a noise-free latent variable z0 is first given, and then the diffusion process is performed step by step. After performing t steps of diffusion, the noisy latent variable z t :

[0062] z t =z t-1 +∈=z0+∈0+∈1+…+∈ (1)

[0063] Among them, ∈0 is the noise added in step 1, ∈1 is the noise added in step 2, and ∈ is the noise added in step t. However, this is inefficient because it requires multiple sampling calculations to obtain z t .

[0064] Advantageously, in the second diffusion process, we can use the derivation of DDPM reparameterization to obtain z directly from z0 by sampling the noise only once. t :

[0065]

[0066] Where, ∈ is the noise sampled from Gaussian distribution, is a preset constant and becomes smaller and smaller as t increases. As t increases, z becomes smaller and smaller. t The content of the original image z0 contained in becomes less and less, while the noise ∈ becomes more and more.

[0067] The above briefly introduces the input latent variable z of the neural network 100 t The generation process of . The construction of the loss function L of the neural network 100 is introduced below.

[0068] Since the unmasked portion of the masked image block is also very different from the original image or line drawing image after rendering, the neural network 100 can accurately predict the added noise L LDM In addition, it is also necessary to use the predicted noise to accurately reconstruct the image L 图像重建To this end, the loss function L of the neural network 100 is composed of two parts as follows:

[0069] L=L LDM +L 图像重建 (3)

[0070]

[0071]

[0072] Among them, z0 represents the hidden variable corresponding to the image without noise, t represents the time step of random sampling, and c t Indicates text condition (text prompt condition), c f Indicates the line draft condition, c s represents the mask image condition, represents the noise ∈,∈ sampled from a Gaussian distribution with mean 0 and variance 1 θ (z t ,t,c t ,c f ,c s ) represents the estimated z t The noise contained in , and z(∈ θ (z t ,t,c t ,c f ,c s )) represents the latent variable of the image reconstructed after removing noise.

[0073] The network parameter that needs to be optimized in the neural network 100 is θ. As mentioned above, the network parameter θ that needs to be optimized involves the Unet encoders 13 and 14, and optionally also involves the Unet decoder 21.

[0074] It should be understood that optimizing the network parameters θ includes calculating the gradient of the network parameters according to the loss function L Then update the parameter θ according to the stochastic gradient descent algorithm: Where μ is the preset learning rate.

[0075] By adjusting the network parameters, the neural network 100 is trained until convergence (ie, the loss function no longer decreases) or a preset number of training rounds is reached.

[0076] It should be noted that the parameters of the Unet encoders 13 and 14 can be directly obtained from the pre-trained Unet encoders 11 and 12.

[0077] It should also be understood that Figure 1 The neural network 100 also includes a basic transformation block (BasicTransformerBlock) (not shown), and the basic transformation block can be considered to be included in the Unet decoder 21.

[0078] Combine the following Figure 1 The neural network 100 is shown to describe Figure 4 A method 400 for training a neural network according to an embodiment of the present invention is shown.

[0079] First, in step 401, a noise image is obtained by adding different noises to the original image. Specifically, in this embodiment, Figure 3 The LDM 300 shown converts the original image into a noise image and then obtains the latent variable z of the noise image. t .

[0080] Next, in step 402, the noise image is input into the first U-shaped network encoder, the line drawing image corresponding to the original image and the noise image are input into the second U-shaped network encoder after being added through the first adder, and the noise image and the line drawing image after zero convolution are input into the third U-shaped network encoder after being added through the second adder.

[0081] Specifically, in this embodiment, the latent variable z of the noise image is t Input Unet encoder 11, and transform the latent variables z of the line drawing image and the noise image into t After being added by the adder 22, the image is input into the Unet encoder 12, and the line drawing image is zero-convolved by the zero-convolution module 25, and then the latent variable z of the noise image is t The signals are added by the adder 23 and then input to the Unet encoder 13 .

[0082] It should be noted that the line drawing image is encoded by an encoder (not shown) that participates in the training of the neural network 100 before being input to the adder 22 and the zero convolution module 25.

[0083] Next, in step 403, the noise image and the partially masked image corresponding to the original image are added by the third adder and then input into the fourth U-shaped network encoder. Specifically, the latent variable z of the masked image and the noise image is t The signals are added by the adder 24 and then input to the Unet encoder 14 .

[0084] It should be noted that the mask image is encoded by a pre-trained encoder (not shown) before being input to the adder 24.

[0085] It should be understood that any part of the original image can be masked with a pure color as needed, preferably with black.

[0086] Finally, in step 404, the neural network is trained. Specifically, in this embodiment, when the loss function L of formulas (3) to (5) is used to train the neural network 100, the parameters of the Unet encoders 13, 14 and the zero convolution modules 25, 26 can be adjusted, while the parameters of the Unet decoder 21, the text encoder 10, the time encoder 20, and the Unet encoders 11, 12 are kept unchanged, so that the neural network 100 converges.

[0087] Optionally, when training the neural network 100, the respective parameters of the Unet encoders 13, 14, the Unet decoder 21 and the zero convolution modules 25, 26 can be adjusted while keeping the respective parameters of the text encoder 10, the time encoder 20, and the Unet encoders 11, 12 unchanged, so that the neural network 100 converges.

[0088] It should also be noted that during the training of the neural network 100, the noise image, the masked image, and the line art image correspond to the same original image.

[0089] It should also be noted that the output result of the Unet encoder 13 is provided to the second input of the Unet decoder 21 after being zero-convolved by the zero-convolution module 26 .

[0090] According to a preferred embodiment, the Unet encoders 11, 12, 13, 14 and the Unet decoder 21 are pre-trained before training the neural network 100.

[0091] Combine the following Figure 1 The neural network 100 is shown to describe Figure 5 A method 500 for performing reasoning using a pre-trained neural network according to an embodiment of the present invention is shown.

[0092] First, in step 501 , the line drawing image is divided into N image blocks, so that adjacent image blocks have overlapping parts, where N is an integer greater than or equal to 2.

[0093] For example, Figure 6 As shown, the long line drawing image to be input into the neural network 100 can be divided into three image blocks with overlapping parts.

[0094] Next, in step 502, the first noise image among the N noise images is input into the first U-shaped network encoder, the first noise image and the first image block of the line draft image are added through the first adder and then input into the second U-shaped network encoder, the first noise image and the convolved first image block of the line draft image are added through the second adder and then input into the third U-shaped network encoder, and the first noise image and the pure color image are added through the third adder and then input into the fourth U-shaped network encoder.

[0095] Specifically, in this embodiment, the latent variable z of the first noise image is t Input Unet encoder 11, the latent variable z of the first noise image t The first image block of the line draft image is added to the first image block of the line draft image by the adder 22 and input to the Unet encoder 12. The first image block of the line draft image is convolved with the latent variable z of the first noise image by the zero convolution module 25. t The two are added by adder 23 and then input into Unet encoder 13, and the latent variable z of the first noise image is t The pure color image is added to the image through the adder 24 and then input into the Unet encoder 14.

[0096] It should be noted that, in this embodiment, the first to Nth noise images can be obtained by directly sampling Gaussian noise and then Figure 3 The LDM 300 shown obtains the corresponding latent variable z t As input.

[0097] It should also be noted that each image block of the line drawing image is encoded by an encoder (not shown) that participates in the training of the neural network 100 before being input into the adder 22 and the zero convolution module 25.

[0098] It should also be noted that the solid color image is encoded by a pre-trained encoder (not shown) before being input to the adder 24.

[0099] It should be understood that Figure 1 The parameters of the zero convolution modules 25, 26 in the neural network 100 shown are not zero during the inference phase and therefore function only as convolution modules.

[0100] Next, in step 503, a first masked image block is obtained based on a first output image block corresponding to the first image block of the line draft image outputted from the U-net decoder. Specifically, in this embodiment, the Unet decoder 21 outputs a first output image block corresponding to the first image block of the line draft image. The first masked image block can be obtained after partially masking the first output image block.

[0101] It should be understood that any part of the output image block of the Unet decoder 21 can be masked with a solid color, preferably black, as needed. For example, if the segmented line drawing image is vertically long, the lower part of the output image block can be masked, and if the segmented line drawing image is horizontally long, the right part of the output image block can be masked.

[0102] It should also be noted that the colors of the solid color image and the colors used for the mask image are the same as the mask colors used when training the neural network 100.

[0103] It should also be noted that the line art image blocks, mask image blocks, and solid color images are all the same size.

[0104] Next, in step 504, the second image block to the Nth image block of the line draft image are sequentially added with the second noise image to the Nth noise image through the first adder and input into the second U-shaped network encoder, and the second image block to the Nth image block of the line draft image are convolved and sequentially added with the second noise image to the Nth noise image through the second adder and input into the third U-shaped network encoder.

[0105] Specifically, in this embodiment, the second image block to the Nth image block of the line draft image are sequentially associated with the latent variables z of the second noise image to the Nth noise image. t After being added by the adder 22, the image blocks from the second image block to the Nth image block of the line drawing image are input into the Unet encoder 12, and the image blocks from the second image block to the Nth image block are convolved by the zero convolution module 23, and then sequentially convolved with the latent variables z of the second noise image to the Nth noise image. t After being added by the adder 23 , the result is input to the Unet encoder 13 .

[0106] Next, in step 505, the first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through a third adder, and then input into a fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image.

[0107] Specifically, in this embodiment, the first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through the adder 24, and then input into the Unet encoder 14 to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image.

[0108] It should be noted that the first mask image block to the N-1th mask image block are obtained in the following manner: the overlapping portions of the first output image block to the N-1th output image block outputted from the Unet encoder 21 and the second image block to the Nth image block of the line drawing image are respectively extracted as a part of the first to the N-1th mask image blocks, and the other part of the first to the N-1th mask image blocks is filled with a solid color.

[0109] Finally, in step 506, the N output image blocks output from the U-net decoder are sequentially spliced ​​into a complete image. Specifically, in this embodiment, the N output image blocks output from the Unet decoder 21 are spliced ​​together to form a complete image.

[0110] Figure 6It is schematically shown that a mask image block is generated from the segmented line drawing image block, and finally a complete image is generated after splicing. In the case of using the neural network 100 for reasoning, the rendered first image block of the line drawing image output from the Unet decoder 21 is obtained based on the first image block of the line drawing image and the solid color image, and the first mask image block and the second mask image block are respectively obtained by masking the lower part of the second output image and the third output image output from the Unet decoder 21. After splicing the first output image block to the third output image block together, the neural network 100 is output.

[0111] Preferably, in this embodiment, weighted fusion can be used for splicing to further improve the continuity problem between output image blocks. Figure 7 Let's describe weighted fusion stitching in detail.

[0112] The principle of weighted fusion stitching is that the closer the overlapping part is to one end of the output image block, the higher the similarity with the end, while the farther the overlapping part is from the end, the higher the similarity with the other end of the output image block. For example, weighted fusion can be achieved by the following formula:

[0113]

[0114] Among them, P1 and P2 represent the overlapping parts of the two output image blocks to be spliced, H represents the height of the overlapping part, and h represents a matrix with the same shape as P1 and P2: for each pixel in the overlapping part, the corresponding element in h represents the vertical distance from the pixel to the lower boundary of the overlapping part, such as Figure 7 shown.

[0115] pass Figure 4 Methods for training neural networks 400 and Figure 5 The method 500 of using a pre-trained neural network for reasoning greatly alleviates the continuity problem between adjacent image blocks, thereby generating a high-quality, consistently colored stitched image without changing any content and details of the stitched image.

[0116] The methods discussed above can be implemented entirely by a computer executable program, or partially or completely by hardware and / or firmware. When implemented by hardware and / or firmware, or when a computer executable program is loaded into a hardware device that can run the program, the apparatus for training a neural network and the apparatus for reasoning using a pre-trained neural network described below are implemented. Below, an overview of these apparatuses is given without repeating some of the details discussed above, but it should be noted that although these apparatuses can perform the methods described above, the methods do not necessarily use those components of the described apparatuses or are not necessarily performed by those components.

[0117] Figure 8 An apparatus 800 for training a neural network according to an embodiment of the present disclosure is shown. The apparatus 800 includes an obtaining unit 801, an input unit 802, and a training unit 803. The obtaining unit 801 obtains a noise image by adding different noises to an original image. The input unit 802 is used to input the noise image into a first U-shaped network encoder, input the line draft image corresponding to the original image and the noise image into a second U-shaped network encoder after adding them through a first adder, input the noise image and the line draft image into a third U-shaped network encoder after adding them through a second adder, and input the noise image and the partially masked image corresponding to the original image into a fourth U-shaped network encoder after adding them through a third adder. The training unit 803 is used to train the neural network in the following manner: adjusting the parameters of the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges; or adjusting the parameters of the U-shaped network decoder, the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

[0118] Figure 8 The apparatus 800 for training a neural network corresponds to Figure 4 Therefore, the relevant details of each unit in the apparatus 800 for training a neural network have been described in detail. Figure 4 The method 400 for training a neural network is described in detail and will not be repeated here.

[0119] Fig. 9An apparatus 900 for performing reasoning using a pre-trained neural network according to an embodiment of the present disclosure is shown. The apparatus 900 includes an image segmentation unit 901, an inference unit 902, and a stitching unit 903. The image segmentation unit 901 is used to segment a line draft image into N image blocks, where adjacent image blocks in the N image blocks have overlapping portions, where N is an integer greater than or equal to 2. The inference unit 902 is used to: input the first noise image among the N noise images into the first U-shaped network encoder, add the first noise image and the first image block among the N image blocks of the line draft image through the first adder and the second adder respectively, and then input them into the second U-shaped network encoder and the third U-shaped network encoder respectively, add the first noise image and the pure color image through the third adder, and then input them into the fourth U-shaped network encoder, obtain the first mask image block based on the first output image block corresponding to the first image block of the line draft image output from the U-shaped network decoder, and sequentially add the second image block to the Nth image block of the line draft image with the second noise image to the Nth noise image among the N noise images through the first adder and the second adder respectively. After addition, the first mask image block to the N-1 mask image block are respectively input to the second U-shaped network encoder and the third U-shaped network encoder, and the first mask image block to the N-1 mask image block are sequentially added with the second noise image to the N noise image through the third adder, and then input to the fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line draft image, wherein the first output image block to the N-1 output image block outputted from the U-shaped network encoder and the overlapping parts of the second image block to the N image block of the line draft image are respectively extracted as a part of the first to the N-1 mask image blocks, and the other part of the first to the N-1 mask image blocks is filled with a solid color, thereby obtaining the first mask image block to the N-1 mask image block. The splicing unit 903 is used to sequentially splice the N output image blocks outputted from the U-shaped network decoder into a complete image.

[0120] Fig. 9 The apparatus 900 for performing reasoning using a pre-trained neural network corresponds to Figure 5 The method 500 for performing reasoning using a pre-trained neural network is shown. Therefore, the relevant details of each unit in the apparatus 900 for performing reasoning using a pre-trained neural network have been described in detail. Figure 5 The description of the method 500 for performing reasoning using a pre-trained neural network is given in detail and will not be repeated here.

[0121] Each component module or unit in the above device can be configured by software, firmware, hardware or a combination thereof. The specific means or methods that can be used for configuration are well known to those skilled in the art and will not be described in detail here. In the case of software or firmware implementation, the data is transmitted from a storage medium or a network to a computer (e.g., a computer) with a dedicated hardware structure. Fig.10 The general-purpose computer 200 shown in the figure is installed with the programs constituting the software, and when the various programs are installed, the computer can execute various functions, etc.

[0122] Fig.10 is a block diagram of an exemplary structure of a general-purpose personal computer in which the methods and / or devices according to embodiments of the present invention may be implemented. Fig.10 As shown, a central processing unit (CPU) 201 performs various processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage device 208 to a random access memory (RAM) 203. In the RAM 203, data required when the CPU 201 performs various processes and the like is also stored as needed. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output interface 205 is also connected to the bus 204.

[0123] The following components are connected to the input / output interface 205: an input device 206 (including a keyboard, a mouse, etc.), an output device 207 (including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage device 208 (including a hard disk, etc.), a communication device 209 (including a network interface card such as a LAN card, a modem, etc.). The communication device 209 performs communication processing via a network such as the Internet. A drive 210 may also be connected to the input / output interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is installed on the drive 210 as needed, so that a computer program read therefrom is installed in the storage device 208 as needed.

[0124] In the case where the above-described series of processing is realized by software, a program constituting the software is installed from a network such as the Internet or a storage medium such as the removable medium 211 .

[0125] It should be understood by those skilled in the art that such storage media is not limited to Fig.10 The removable medium 211 shown has a program stored therein and is distributed separately from the device to provide the program to the user. Examples of the removable medium 211 include magnetic disks (including floppy disks (registered trademark)), optical disks (including compact disk read only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including minidiscs (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 202, a hard disk included in the storage device 208, or the like, in which the program is stored and distributed to the user together with the device containing them.

[0126] The present disclosure also provides corresponding computer program codes and a computer program product storing machine-readable instruction codes. When the instruction codes are read and executed by a machine, the above-mentioned methods 400 and 500 according to the implementation of the present disclosure can be executed.

[0127] Accordingly, the storage medium configured to carry the program product storing the machine-readable instruction code is also included in the disclosure of the present disclosure, including but not limited to a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, and the like.

[0128] Through the above description, the embodiments of the present disclosure provide the following technical solutions, but are not limited thereto.

[0129] Note 1. A device for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder and a third adder, the output end of the first U-shaped network encoder is connected to the first input end of the U-shaped network decoder, the first adder is connected to the input end of the second U-shaped network encoder, the second adder is connected to the input end of the third U-shaped network encoder, the third adder is connected to the input end of the fourth U-shaped network encoder, and the output end of the second U-shaped network encoder, the output end of the third U-shaped network encoder and the output end of the fourth U-shaped network encoder are connected to the second input end of the U-shaped network decoder, the device comprising:

[0130] an obtaining unit configured to obtain a noise image by adding different noises to the original image;

[0131] An input unit is configured to:

[0132] The noisy image is input into the first U-shaped network encoder,

[0133] The line drawing image and the noise image corresponding to the original image are added by the first adder and then input into the second U-shaped network encoder,

[0134] The noise image and the line drawing image are added together by the second adder and then input into the third U-shaped network encoder, and

[0135] inputting the noise image and the partially masked image corresponding to the original image into a fourth U-shaped network encoder after adding them through a third adder; and

[0136] A training unit is configured to train the neural network by:

[0137] Adjusting the parameters of the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or

[0138] The parameters of the U-shaped network decoder, the third U-shaped network encoder and the fourth U-shaped network encoder are adjusted, and the parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder are kept unchanged, so that the neural network converges.

[0139] Note 2. The device according to Note 1, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder;

[0140] wherein the text encoder is input with prompts, and the temporal encoder is input with timesteps to obtain respective tensors of prompts and timesteps; and

[0141] Among them, the respective tensors of prompts and time steps are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder respectively.

[0142] Note 3. The device according to Note 2, wherein the neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third U-shaped network encoder and the second input end of the U-shaped network decoder;

[0143] The training unit is further configured to adjust parameters of the first zero convolution module and parameters of the second zero convolution module during training; and

[0144] The line drawing image is provided to the second adder after being zero-convolved by the first zero-convolution module.

[0145] Note 4. An apparatus according to Note 3, wherein the total loss function of the neural network is constructed as the sum of a first loss function and a second loss function, wherein the first loss function is a square sum loss function between randomly sampled noise and noise estimated for a noise image, and the second loss function is a square sum loss function between latent variables corresponding to an original image without noise and latent variables of a reconstructed image after noise removal.

[0146] Note 5. An apparatus according to any one of Notes 1 to 4, wherein, before the noise image is input into the first U-shaped network encoder, the first adder, the second adder and the third adder, the latent variables of the original image are obtained by another pre-trained encoder, and then noise is added to the latent variables to obtain the latent variables corresponding to the noise image.

[0147] Note 6. An apparatus according to Note 3, wherein the line drawing image is encoded by another encoder participating in the training of the neural network before being input into the first adder and the first zero convolution module.

[0148] Note 7. An apparatus according to any one of Notes 1 to 4, wherein the partially masked image is encoded by a further encoder participating in the training of a neural network before being input into the third adder.

[0149] Note 8. An apparatus according to any one of Notes 1 to 5, wherein the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder are pre-trained before training the neural network.

[0150] Note 9. A device for performing reasoning using a pre-trained neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder, and a third adder, the output end of the first U-shaped network encoder is connected to the first input end of the U-shaped network decoder, the first adder is connected to the input end of the second U-shaped network encoder, the second adder is connected to the input end of the third U-shaped network encoder, the third adder is connected to the input end of the fourth U-shaped network encoder, and the output end of the second U-shaped network encoder, the output end of the third U-shaped network encoder, and the output end of the fourth U-shaped network encoder are connected to the second input end of the U-shaped network decoder, the device comprising:

[0151] An image segmentation unit, which is configured to:

[0152] Divide the line drawing image into N image blocks, where adjacent image blocks in the N image blocks have overlapping parts, where N is an integer greater than or equal to 2; and

[0153] An inference unit configured to perform the following operations:

[0154] Inputting the first noise image of the N noise images into the first U-shaped network encoder,

[0155] The first image block of the N image blocks of the first noise image and the line drawing image is added by the first adder and the second adder respectively, and then input into the second U-shaped network encoder and the third U-shaped network encoder respectively,

[0156] The first noise image and the pure color image are added by the third adder and then input into the fourth U-shaped network encoder.

[0157] A first mask image block is obtained based on a first output image block corresponding to a first image block of the line drawing image outputted from the U-shaped network decoder,

[0158] The second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image in the N noise images through the first adder and the second adder, and then input into the second U-shaped network encoder and the third U-shaped network encoder respectively.

[0159] The first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through a third adder, and then input into a fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line draft image.

[0160] Among them, the first to N-1th masked image blocks are obtained by extracting the overlapping parts of the first output image block to the N-1th output image block outputted from the U-shaped network encoder and the second image block to the N-1th image block of the line drawing image as a part of the first to N-1th masked image blocks, and filling the other part of the first to N-1th masked image blocks with a solid color.

[0161] Note 10. The device according to Note 9 further comprises a splicing unit configured to sequentially splice N output image blocks output from the U-shaped network decoder into a complete image,

[0162] The splicing unit is further configured to splice the N output image blocks in a weighted fusion manner, so that parts of the N output image blocks closer to the edge have higher weights.

[0163] Note 11. The device according to Note 9, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder;

[0164] wherein the text encoder is input with prompts, and the temporal encoder is input with timesteps to obtain respective tensors of prompts and timesteps; and

[0165] Among them, the respective tensors of prompts and time steps are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder respectively.

[0166] Note 12. The device according to Note 9, wherein the neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third U-shaped network encoder and the second input end of the U-shaped network decoder,

[0167] wherein the first zero convolution module and the second zero convolution module perform convolutions with non-zero parameters during inference of the neural network, and

[0168] The N image blocks of the line drawing image are provided to the second adder after being convolved by the first zero convolution module.

[0169] Note 13. A method for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder and a third adder, the output end of the first U-shaped network encoder is connected to the first input end of the U-shaped network decoder, the first adder is connected to the input end of the second U-shaped network encoder, the second adder is connected to the input end of the third U-shaped network encoder, the third adder is connected to the input end of the fourth U-shaped network encoder, and the output end of the second U-shaped network encoder, the output end of the third U-shaped network encoder and the output end of the fourth U-shaped network encoder are connected to the second input end of the U-shaped network decoder, the method comprising:

[0170] The noisy image is obtained by adding different noises to the original image;

[0171] Input the noisy image into the first U-shaped network encoder;

[0172] The line drawing image and the noise image corresponding to the original image are added by a first adder and then input into a second U-shaped network encoder;

[0173] The noise image and the line drawing image are added by the second adder and then input into the third U-shaped network encoder;

[0174] inputting the noise image and the partially masked image corresponding to the original image into a fourth U-shaped network encoder after adding them through a third adder; and

[0175] The neural network is trained as follows:

[0176] Adjusting the parameters of the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or

[0177] The parameters of the U-shaped network decoder, the third U-shaped network encoder and the fourth U-shaped network encoder are adjusted, and the parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder are kept unchanged, so that the neural network converges.

[0178] Note 14. The method according to Note 13, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder; and

[0179] The neural network also includes a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third U-shaped network encoder and the second input end of the U-shaped network decoder, wherein training the neural network also includes adjusting the parameters of the first zero convolution module and the parameters of the second zero convolution module.

[0180] Note 15. A method according to Note 14, wherein the total loss function of the neural network is constructed as the sum of a first loss function and a second loss function, wherein the first loss function is a square sum loss function between randomly sampled noise and noise estimated for a noisy image, and the second loss function is a square sum loss function between latent variables corresponding to an original image without noise and latent variables of a reconstructed image after noise removal.

[0181] Note 16. A method for reasoning using a pre-trained neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder and a third adder, the output end of the first U-shaped network encoder is connected to the first input end of the U-shaped network decoder, the first adder is connected to the input end of the second U-shaped network encoder, the second adder is connected to the input end of the third U-shaped network encoder, the third adder is connected to the input end of the fourth U-shaped network encoder, and the output end of the second U-shaped network encoder, the output end of the third U-shaped network encoder and the output end of the fourth U-shaped network encoder are connected to the second input end of the U-shaped network decoder, the method comprising:

[0182] The line drawing image is divided into N image blocks, where adjacent image blocks in the N image blocks have overlapping parts, where N is an integer greater than or equal to 2;

[0183] Inputting a first noise image among the N noise images into a first U-shaped network encoder;

[0184] After adding the first image block of the N image blocks of the first noise image and the line drawing image through the first adder and the second adder respectively, the first image block is input into the second U-shaped network encoder and the third U-shaped network encoder respectively;

[0185] The first noise image and the pure color image are added by a third adder and then input into a fourth U-shaped network encoder;

[0186] Obtaining a first mask image block based on a first output image block output from the U-shaped network decoder corresponding to a first image block of the line drawing image;

[0187] Adding the second image block to the Nth image block of the line drawing image to the second noise image to the Nth noise image in sequence through a first adder and a second adder respectively, and inputting the resultant to a second U-shaped network encoder and a third U-shaped network encoder respectively; and

[0188] The first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through a third adder, and then input into a fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line draft image.

[0189] Among them, the first to N-1th masked image blocks are obtained by extracting the overlapping parts of the first output image block to the N-1th output image block outputted from the U-shaped network encoder and the second image block to the N-1th image block of the line drawing image as a part of the first to N-1th masked image blocks, and filling the other part of the first to N-1th masked image blocks with a solid color.

[0190] Note 17. The method according to Note 16 also includes using weighted fusion to sequentially splice the N output image blocks output from the U-shaped network decoder into a complete image, so that the parts of the N output image blocks closer to the edge have higher weights.

[0191] Note 18. The method according to Note 16 or 17, wherein the text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder, wherein the text encoder is input with a prompt, and the time encoder is input with a time step to obtain respective tensors of the prompt and the time step, so as to provide them to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder respectively; and

[0192] The neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output end of the third U-shaped network encoder and the second input end of the U-shaped network decoder, wherein the first zero convolution module and the second zero convolution module perform convolutions with non-zero parameters during inference of the neural network.

[0193] Note 19. A computer-readable storage medium having a program stored thereon, the program causing a computer to perform a method according to any one of Notes 13 to 15 when executed by a processor.

[0194] Note 20. A computer-readable storage medium having a program stored thereon, the program causing a computer to perform a method according to any one of Notes 16 to 18 when executed by a processor.

[0195] Finally, it should be noted that the terms "comprises", "includes" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article or device. In addition, in the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the presence of other identical elements in the process, method, article or device that includes the elements.

[0196] Although the embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings, it should be understood that the embodiments described above are only configured to illustrate the present disclosure and do not constitute a limitation on the present disclosure. For those skilled in the art, various modifications and changes can be made to the above embodiments without departing from the essence and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the attached claims and their equivalents.

Claims

1. A device for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder and a third adder, the output end of the first U-shaped network encoder is connected to the first input end of the U-shaped network decoder, the first adder is connected to the input end of the second U-shaped network encoder, the second adder is connected to the input end of the third U-shaped network encoder, the third adder is connected to the input end of the fourth U-shaped network encoder, and the output end of the second U-shaped network encoder, the output end of the third U-shaped network encoder and the output end of the fourth U-shaped network encoder are connected to the second input end of the U-shaped network decoder, the device comprising: an obtaining unit configured to obtain a noise image by adding different noises to the original image; An input unit is configured to: Inputting the noise image into the first U-shaped network encoder, The line drawing image corresponding to the original image and the noise image are added by the first adder and then input into the second U-shaped network encoder, The noise image and the line drawing image are added by the second adder and then input into the third U-shaped network encoder, and inputting the noise image and the partially masked image corresponding to the original image into the fourth U-shaped network encoder after adding them through the third adder; and A training unit is configured to train the neural network by: adjusting the parameters of the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or Adjust the respective parameters of the U-shaped network decoder, the third U-shaped network encoder and the fourth U-shaped network encoder, and keep the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

2. The device according to claim 1, wherein: The text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder; wherein the text encoder is input with a prompt, and the temporal encoder is input with a time step to obtain a tensor of the prompt and the time step respectively; and Wherein, the respective tensors of the prompt and the time step are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder respectively.

3. The device according to claim 2, wherein: The neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between an output terminal of the third U-shaped network encoder and a second input terminal of the U-shaped network decoder; as well as The line drawing image is provided to the second adder after being zero-convolved by the first zero-convolution module.

4. The device according to any one of claims 1 to 3, wherein: The total loss function of the neural network is constructed as the sum of a first loss function and a second loss function, wherein the first loss function is a square sum loss function between randomly sampled noise and noise estimated for the noisy image, and the second loss function is a square sum loss function between latent variables corresponding to the original image without noise and latent variables of the reconstructed image after noise removal.

5. A device for performing reasoning using a pre-trained neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder, and a third adder, the output of the first U-shaped network encoder is connected to the first input of the U-shaped network decoder, the first adder is connected to the input of the second U-shaped network encoder, the second adder is connected to the input of the third U-shaped network encoder, the third adder is connected to the input of the fourth U-shaped network encoder, and the output of the second U-shaped network encoder, the output of the third U-shaped network encoder, and the output of the fourth U-shaped network encoder are connected to the second input of the U-shaped network decoder, the device comprising: An image segmentation unit, which is configured to: Divide the line draft image into N image blocks, wherein adjacent image blocks in the N image blocks have overlapping parts, where N is an integer greater than or equal to 2; as well as An inference unit configured to perform the following operations: Inputting a first noise image among the N noise images into the first U-shaped network encoder, After adding the first image block of the first noise image and the first image block of the N image blocks of the line drawing image through the first adder and the second adder respectively, the first image block is input into the second U-shaped network encoder and the third U-shaped network encoder respectively. The first noise image and the pure color image are added by the third adder and then input into the fourth U-shaped network encoder, obtaining a first mask image block based on a first output image block corresponding to a first image block of the line drawing image outputted from the U-shaped network decoder, The second image block to the Nth image block of the line drawing image are sequentially added to the second noise image to the Nth noise image in the N noise images through the first adder and the second adder respectively, and then input into the second U-shaped network encoder and the third U-shaped network encoder respectively, The first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through the third adder, and then input into the fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image, Among them, the first to the N-1th masked image blocks are obtained by extracting the overlapping parts of the first output image block to the N-1th output image block outputted from the U-shaped network encoder and the second image block to the N-1th image block of the line drawing image as a part of the first to the N-1th masked image blocks, and filling the other part of the first to the N-1th masked image blocks with a solid color.

6. The apparatus according to claim 5, further comprising a splicing unit configured to sequentially splice the N output image blocks output from the U-shaped network decoder into a complete image.

7. The device according to claim 5, wherein: The text encoder and the time encoder are connected to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder; wherein the text encoder is input with a prompt, and the temporal encoder is input with a time step to obtain a tensor of the prompt and the time step respectively; and Wherein, the respective tensors of the prompt and the time step are provided to the first U-shaped network encoder, the second U-shaped network encoder, the third U-shaped network encoder, the fourth U-shaped network encoder and the U-shaped network decoder respectively.

8. The device according to any one of claims 5 to 7, wherein: The neural network further comprises a first zero convolution module and a second zero convolution module, the first zero convolution module is connected to the second adder, and the second zero convolution module is connected between the output of the third U-shaped network encoder and the second input of the U-shaped network decoder, wherein the first zero convolution module and the second zero convolution module perform convolutions with non-zero parameters during inference of the neural network, and The N image blocks of the line drawing image are provided to the second adder after being convolved by the first zero convolution module.

9. A computer-readable storage medium having a program stored thereon, the program, when executed by a processor, causing a computer to perform a method for training a neural network, the neural network comprising a text encoder, a time encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder, and a third adder, the output of the first U-shaped network encoder being connected to the first input of the U-shaped network decoder, the first adder being connected to the input of the second U-shaped network encoder, the second adder being connected to the input of the third U-shaped network encoder, the third adder being connected to the input of the fourth U-shaped network encoder, and the output of the second U-shaped network encoder, the output of the third U-shaped network encoder, and the output of the fourth U-shaped network encoder being connected to the second input of the U-shaped network decoder, the method comprising: The noisy image is obtained by adding different noises to the original image; Inputting the noise image into the first U-shaped network encoder; Adding the line drawing image corresponding to the original image and the noise image through the first adder and inputting them into the second U-shaped network encoder; Adding the noise image and the line drawing image through the second adder and then inputting them into the third U-shaped network encoder; inputting the noise image and the partially masked image corresponding to the original image into the fourth U-shaped network encoder after adding them through the third adder; and The neural network is trained as follows: adjusting the parameters of the third U-shaped network encoder and the fourth U-shaped network encoder, and keeping the parameters of the U-shaped network decoder, the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges, or Adjust the respective parameters of the U-shaped network decoder, the third U-shaped network encoder and the fourth U-shaped network encoder, and keep the respective parameters of the text encoder, the time encoder, the first U-shaped network encoder and the second U-shaped network encoder unchanged, so that the neural network converges.

10. A computer-readable storage medium having a program stored thereon, the program, when executed by a processor, causing a computer to perform a method for performing inference using a pre-trained neural network, the neural network comprising a text encoder, a temporal encoder, a first U-shaped network encoder, a second U-shaped network encoder, a third U-shaped network encoder, a fourth U-shaped network encoder, a U-shaped network decoder, a first adder, a second adder, and a third adder, the output of the first U-shaped network encoder being connected to the first input of the U-shaped network decoder, the first adder being connected to the input of the second U-shaped network encoder, the second adder being connected to the input of the third U-shaped network encoder, the third adder being connected to the input of the fourth U-shaped network encoder, and the output of the second U-shaped network encoder, the output of the third U-shaped network encoder, and the output of the fourth U-shaped network encoder being connected to the second input of the U-shaped network decoder, the method comprising: Divide the line draft image into N image blocks, wherein adjacent image blocks in the N image blocks have overlapping parts, where N is an integer greater than or equal to 2; Inputting a first noise image among the N noise images into the first U-shaped network encoder; Adding the first image block of the N image blocks of the first noise image and the line drawing image through the first adder and the second adder respectively, and then inputting the first image block into the second U-shaped network encoder and the third U-shaped network encoder respectively; Adding the first noise image and the pure color image through the third adder, and then inputting the resultant to the fourth U-shaped network encoder; Obtaining a first mask image block based on a first output image block output from the U-shaped network decoder corresponding to a first image block of the line drawing image; Adding the second image block to the Nth image block of the line drawing image to the second noise image to the Nth noise image in the N noise images in sequence through the first adder and the second adder respectively, and then inputting the two images into the second U-shaped network encoder and the third U-shaped network encoder respectively; and The first mask image block to the N-1th mask image block are sequentially added with the second noise image to the Nth noise image through the third adder, and then input into the fourth U-shaped network encoder to sequentially obtain N output image blocks corresponding to the N image blocks of the line drawing image, Among them, the first to the N-1th masked image blocks are obtained by extracting the overlapping parts of the first output image block to the N-1th output image block outputted from the U-shaped network encoder and the second image block to the N-1th image block of the line drawing image as a part of the first to the N-1th masked image blocks, and filling the other part of the first to the N-1th masked image blocks with a solid color.

Images