Image processing apparatus, image processing method, and program

Abstract

To provide an image processing apparatus, an image processing method, and a program capable of improving coloring accuracy in image conversion for coloring an image with a small amount of information, such as a line drawing.SOLUTION: An image processing apparatus comprises an image processing unit including a first multilayer neural network for converting RGB images to Latent images and vice versa and a second multilayer neural network for performing a de-diffusion process on the Latent image. The image processing unit, when decoding the Latent image, on which the de-diffusion process has been performed in the second multilayer neural network, into the RGB image in the first multilayer neural network, uses the Latent image or an image feature amount and a condition image resized to the same size, as input to an input layer and an intermediate layer of the first multilayer neural network.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] In recent years, advances in AI (Artificial Intelligence) technology have led to the emergence of numerous applications for generating and converting images. These applications allow even those without photography, drawing, or retouching skills to easily generate and edit images. Image generation AI, a technology that learns from a large number of images and generates new images, can be broadly categorized into three types of generative models: VAE (Variational Auto Encoder), GAN (Generative Adversarial Network), and diffusion models. Diffusion models produce high-quality images that are indistinguishable from real photographs or human-created drawings and illustrations. A common technique for diffusion models is text-to-image, which involves inputting a string of characters (prompt) specifying the image to be generated and outputting the image.

[0003] A diffusion model learns a diffusion process that gradually adds noise to an image and a dediffusion process (or generation process) that gradually removes noise from the image. A new image is generated from the noise by inputting noise (latent variables) generated using random numbers and calculating the dediffusion process using a trained model to remove the noise. Non-Patent Document 1 below discloses a latent diffusion model (LDM), which is one type of diffusion model. In addition to a conventional diffusion model, the LDM includes an encoder / decoder that can convert between image space and latent space. Unlike conventional diffusion models, which calculate the diffusion process and dediffusion process in image space, the LDM calculates the diffusion process and dediffusion process in latent space. Furthermore, by inputting text or images as conditions when calculating the dediffusion process, it is possible to generate or convert images taking the conditions into account. Furthermore, by using images as conditions, the LDM enables a variety of image transformations, such as coloring images, generating depth images from RGB images, and the inverse transformation of these. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models”, Computer Vision and Pattern Recognition(cs.CV), 13 April 2022 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Non-Patent Document 1, when a source image is input as a condition in image conversion, the image size of the source image is downsampled using interpolation to the same size as the latent image. Because this downsampling causes information in the source image to be lost, conversion is not performed correctly for images with fine structures (textures), such as photographs and line drawings.

[0006] In view of the above-mentioned problems, an object of the present invention is to provide an image processing device, an image processing method, and a program that can improve coloring accuracy in image conversion for coloring images with little information, such as line drawings. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, an image processing device according to one embodiment of the present invention is an image processing device that includes an image processing unit having a first multilayer neural network that converts an RGB image and a Latent image into each other, and a second multilayer neural network that performs a de-diffusion process on the Latent image, and when decoding the Latent image, for which the de-diffusion process has been performed in the second multilayer neural network, into the RGB image in the first multilayer neural network, the image processing unit uses the Latent image or image features together with a condition image resized to the same size as the Latent image as input to an input layer and an intermediate layer of the first multilayer neural network.

[0008] An image processing method according to one embodiment of the present invention includes an image processing step having a first multilayer neural network that converts an RGB image and a Latent image into each other, and a second multilayer neural network that performs a de-diffusion process on the Latent image, wherein the image processing step is an image processing method executed by a computer, in which when the Latent image, after the de-diffusion process has been performed in the second multilayer neural network, is decoded into the RGB image in the first multilayer neural network, the Latent image or image features and a condition image resized to the same size are used together as inputs to the input layer and intermediate layer of the first multilayer neural network.

[0009] A program according to one embodiment of the present invention causes a computer to function as image processing means having a first multilayer neural network that converts an RGB image and a Latent image into each other, and a second multilayer neural network that performs a de-diffusion process on the Latent image, wherein when the Latent image, which has undergone the de-diffusion process in the second multilayer neural network, is decoded into the RGB image in the first multilayer neural network, the image processing means uses the Latent image or image features, and a condition image resized to the same size, as inputs to the input layer and intermediate layer of the first multilayer neural network. [Effects of the Invention]

[0010] According to the present invention, it is possible to improve the coloring accuracy in image conversion for coloring an image with a small amount of information, such as a line drawing. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to an embodiment of the present invention. [Figure 3]10A and 10B are diagrams illustrating resizing of a condition image according to the present embodiment. [Figure 4] FIG. 10 is a diagram illustrating an example of the use of Group Normalization according to the present embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of the use of Group Normalization according to the present embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing unit according to the present embodiment. [Figure 7] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing unit according to the present embodiment. [Figure 8] 10 is a flowchart illustrating an example of a processing flow according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0013] <1. Image processing system configuration> The configuration of an image processing system according to this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an image processing system according to this embodiment.

[0014] The image processing system 1 shown in FIG. 1 is a system for automatically coloring line drawings. Images to be colored include realistic images such as photographs, and non-realistic images such as manga and anime. The image processing system 1 is used, for example, to colorize electronic books. Examples of electronic books include content such as books, magazines, manga, and newspapers. This embodiment will be described below using an example in which the image processing system 1 colorizes a monochrome manga.

[0015] 1, the image processing system 1 includes a user terminal 10 and an image processing device 20. The user terminal 10 and the image processing device 20 are connected to each other so as to be able to communicate with each other via a network NW. The network NW may be configured to transmit and receive information using, for example, a LAN (Local Area Network), a WAN (Wide Area Network), a telephone network (such as a mobile phone network or a fixed telephone network), a regional IP (Internet Protocol) network, or the Internet.

[0016] (1) User terminal 10 The user terminal 10 is a terminal owned by a user. The user terminal 10 is, for example, a smartphone, a tablet terminal, or a PC (Personal Computer).

[0017] A screen for coloring a line drawing image is displayed on the user terminal 10 by an application for the user to color a line drawing image (hereinafter also referred to as a "line drawing coloring application") The user colors the line drawing image by operating the screen displayed on the user terminal 10 by the line drawing coloring application.

[0018] The functions of the line drawing coloring application may be provided by installing the line drawing coloring application on the user terminal 10 (i.e., a native application), or may be provided by a web system (i.e., a web application). In the case of a web system, the functions of the line drawing coloring application are provided via a web browser.

[0019] (2) Image processing device 20 The image processing device 20 is a device for coloring a line drawing image. The image processing device 20 is realized, for example, by one or more servers (for example, cloud servers). In the server, various processes are executed by a program for functioning as the image processing device 20.

[0020] The image processing device 20 uses an image generation AI to color a line drawing image. Image generation AI is broadly classified into three types of generative models: VAE (Variational Auto Encoder), GAN (Generative Adversarial Network), and diffusion model. Of these generative models, the image processing device 20 uses LDM (Latent Diffusion Models), which combines VAE and diffusion model, to color a line drawing image.

[0021] DDPM (Denoising Diffusion Probabilistic Models), a predecessor technology of LDM, realized image generation from noise by learning to denoise training data images in RGB space (image space). Furthermore, Palette: Image-to-Image Diffusion Models emerged, which applied DDPM to image conversion by using condition images as inputs during denoising training. LDM does not learn denoising in RGB space, but in latent space (latent space). For example, the image processing device 20 uses a pre-trained VAE to convert the image to be colored into a latent image by encoding it from RGB space to latent space, and then learns denoising for the latent image. This allows the image processing device 20 to generate and convert (color) higher-resolution images using a diffusion model.

[0022] <2. Functional configuration of image processing device> The configuration of the image processing system 1 according to this embodiment has been described above. Next, the functional configuration of the image processing device 20 according to this embodiment will be described with reference to Fig. 2 to Fig. 6. Fig. 2 is a block diagram showing an example of the functional configuration of the image processing device 20 according to this embodiment. As shown in FIG. 1, the image processing device 20 includes a communication unit 210, a storage unit 220, and a control unit 230. Equipped with.

[0023] (1) Communications Unit 210 The communication unit 210 has a function of transmitting and receiving various information. In communication with the user terminal 10 via the network NW, the communication unit 210 receives information corresponding to operations input by the user to the user terminal 10, an original image, a condition image, auxiliary information, etc., and transmits a converted image, etc. The original image is an image before image editing or processing (for example, coloring, color correction, inpainting, etc.) is performed, and is an image to be colored (a line drawing image). The converted image is an image after image editing or processing has been performed, and is an image obtained by coloring the original image (a colored image).

[0024] A condition image is an image that indicates that a subject (e.g., a character or object) in a condition image corresponds to a subject in a transformed image at pixels with the same coordinates. Condition images include, for example, a source image, a semantic image, a pose (bone), a depth, and a low-resolution image. A semantic image is an image to which an identification ID of a subject in a source image is assigned on a pixel-by-pixel basis. A pose (bone) is an image that represents the posture of the subject. A depth is an image that represents the distance to the subject. In this embodiment, at least the source image is used as the condition image, and semantic images, poses, depths, low-resolution images, and the like other than the source image may also be used as auxiliary condition images.

[0025] The auxiliary information is information used in conjunction with the condition image as an auxiliary, such as an auxiliary condition image or a reference image. Unlike the condition image, the reference image is an image in which the subject in the reference image and the subject in the converted image do not correspond to each other at the same pixel coordinates. However, the reference image is assumed to contain the same subject as the converted image. The reference image is, for example, a color image of a different page from the source image, depicting the same character as the source image.

[0026] (2) Storage section 220 The storage unit 220 has a function of storing various types of information. The storage unit 220 is configured by a storage medium provided as hardware in the image processing device 20, such as a hard disk drive (HDD), a solid state drive (SSD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a random access read / write memory (RAM), a read-only memory (ROM), or any combination of these storage media.

[0027] The storage unit 220 stores, for example, data used as training data for training a multilayer neural network used in an image generation AI, such as a source image, a semantic image, a pose, a depth, a low-resolution image, a reference image, and a converted image.

[0028] (3) Control unit 230 The control unit 230 has a function of controlling the overall operation of the image processing device 20. The control unit 230 is realized, for example, by causing a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) provided as hardware in the image processing device 20 to execute a program. As shown in FIG. 2, the control unit 230 includes a learning data generation unit 231, a learning unit 232, an image processing unit 233, and an output control unit 234.

[0029] (3-1) Learning Data Generation Unit 231 The training data generation unit 231 has a function of generating training data for training a multilayer neural network used in the image generation AI. The training data generation unit 231 reads images to be used as training data from the storage unit 220 and forms the training data. The training data is, for example, a set of a condition image, which is an original image, and a converted image. The condition images of the training data may also include auxiliary condition images (auxiliary information) such as semantic images, poses, depth images, and low-resolution images other than the source image. In addition, a reference image may also be used as auxiliary information in addition to the condition image.

[0030] (3-2) Learning Section 232 The learning unit 232 has a function of training the multilayer neural network used in the image generation AI. The learning unit 232 uses a diffusion model, which is one of the supervised learning methods, as the loss function of the multilayer neural network that converts images. The learning unit 232 uses the condition image and the reference image as input data c and r of the multilayer neural network, respectively, and converts the converted image into the noise-mixed image x, which is the input data. t is used as x0 to create the loss function, and the loss is calculated using the following equation (1). For example, the mean square error (MSE) between the estimated value and the target value is used as the loss function.

[0031]

number

[0032] In the above equation (1), ε represents the random noise added to x0, t represents the step in the diffusion model, and ε θ (x0,t,c,r) represents a multilayer neural network that predicts noise at a certain time t in a diffusion model. When training with a diffusion model, the output of the multilayer neural network is not the transformed image, but the noise mixed into the RGB image at time t and the latent image, which is an image feature. Therefore, to obtain the transformed image, it is necessary to perform sampling (denoising) using a sampler such as DDIM (Denoising Diffusion Implicit Models) or DPM Solver with the trained multilayer neural network.

[0033] The LDM includes a first multilayer neural network, which is a VAE that converts images between RGB images belonging to RGB space and latent images belonging to latent space, and a second multilayer neural network, which is a diffusion model that denoises latent images in latent space. After training the first multilayer neural network, the training unit 232 converts the RGB images into latent images and trains the second multilayer neural network.

[0034] The first multilayer neural network performs mutual conversion between RGB images and latent images. During the mutual conversion, the image size of the latent image becomes smaller than that of the RGB image due to convolution and downsampling processing in the encoder part of the first multilayer neural network. Therefore, in training the second multilayer neural network, the training unit 232 performs denoising training using the latent image, which has an image size smaller than that of the RGB image. When performing an image transformation task using LDM, the source image used as the condition image is resized to the same image size as the noisy latent image that is the input of the second multilayer neural network, concatenated along the channel axis, and input to the input layer of the second multilayer neural network.

[0035] When training an image translation task, it is common to prepare training data in which the source and transformed images are the same size. However, when training an image translation task using LDM, the source image must be reduced to match the latent image. If this reduction is performed using a general reduction process that does not change the number of channels, such as reduction using bilinear interpolation, as is done in conventional image processing, the amount of information in the source image is lost, resulting in a decrease in the quality of the image conversion. In particular, high-frequency information such as the texture of photographs and paintings, and the line drawings in illustrations and comics is lost, resulting in the collapse of local structure in the converted image.

[0036] Therefore, the learning unit 232 learns about the case where the condition image, which is the source image, is input not only to the input layer of the second multilayer neural network, but also to the intermediate layers of the second multilayer neural network and to the input layer and intermediate layers of the first multilayer neural network that decodes the latent image into an RGB image by convolution and upsampling.When inputting the condition image to each layer, the learning unit 232 combines resized condition images to reduce loss of information in the condition image and can learn image transformation that does not destroy local structure. In particular, in the decoder section of the first multilayer neural network, the image size of the image features, which are the output or input of the intermediate layer of the multilayer neural network, increases as it approaches the output layer due to the convolution and upsampling described above. Therefore, when the condition image is resized to the same size as the image features for combining, information loss is further reduced.

[0037] (3-3) Image processing unit 233 The image processing unit 233 has a function of coloring a line drawing image using an image generation AI. The image processing unit 233 has a first multilayer neural network that converts between an RGB image and a latent image, and a second multilayer neural network that performs a de-diffusion process on the latent image.

[0038] When the image processing unit 233 decodes the latent image that has undergone the dediffusion process in the second multilayer neural network into an RGB image in the first multilayer neural network, the image processing unit 233 inputs the latent image or image feature as an input to the input layer and intermediate layer of the first multilayer neural network. At this time, the image processing unit 233 may also input a condition image resized to the same size as the latent image or image feature as an input to the input layer and intermediate layer of the first multilayer neural network. This reduces the loss of information in the condition image and prevents local structures from collapsing. For example, when coloring manga, the structure and lines of characters and objects are less likely to collapse. This improves the accuracy of image generation and conversion (coloring).

[0039] Furthermore, when the image processing unit 233 performs the dediffusion process in the second multilayer neural network, it inputs the latent image or image feature as an input to the input layer and intermediate layer of the second multilayer neural network. At this time, the image processing unit 233 may also input a condition image resized to the same size as the latent image or image feature as an input to the input layer and intermediate layer of the second multilayer neural network. This reduces the loss of information in the condition image and prevents local structures from collapsing. For example, when coloring manga, the structure and lines of characters and objects are less likely to collapse. This improves the accuracy of image generation and conversion (coloring).

[0040] Methods for using condition images as input include combining them with latent images or image features on the channel axis, adjusting the number of channels using convolution to input them into Group Normalization after combining, and inputting them without combining them like Cross Attention.

[0041] In addition, the image processing unit 233 divides the image plane of the condition image into patches to match the image size of the multi-layer resolution latent image or image feature processed in the multi-layer neural network, and resizes the condition image by rearranging and combining each pixel belonging to the patch in the channel axis direction. A commonly known method for resizing an image is reduction processing using bilinear interpolation, etc. However, general reduction processing such as reduction processing using bilinear interpolation, which does not change the number of channels, cannot prevent the loss of information in the condition image. Therefore, the image processing unit 233 uses a resizing method that changes the number of channels, unlike general reduction processing, making it possible to prevent loss of information in the condition image due to resizing.

[0042] Here, resizing of the condition image according to this embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining resizing of the condition image according to this embodiment.

[0043] As shown in Fig. 3, the image processing unit 233 first divides the image plane of the condition image into patches according to the image size of the latent image or image feature. The left side of Fig. 3 shows a condition image with a width size of w, a height size of h, and a number of channels of c. The image processing unit 233 divides each of the width size w and the height size h into four, for example. Next, the image processing unit 233 rearranges each pixel belonging to the patch in the channel axis direction. The right side of Fig. 3 shows a condition image with a reduction ratio of r, a width size of w / r, a height size of h / r, and a number of channels of r. 2 The condition image is shown resized to c. When the width w and height h are each divided into four, the image processing unit 233 resizes the condition image so that the width w / 2, height h / 2, and number of channels are 4c, for example, with a reduction ratio r=2. Note that the number of divisions and the numerical values ​​of the reduction ratio are not limited to this example. This prevents the image processing unit 233 from losing information about the condition image due to resizing. Therefore, the image processing unit 233 can combine the condition image with the latent image or image feature amount without losing any information about the condition image, and input the combined image to the input layer and intermediate layer of each multilayer neural network.

[0044] In recent neural networks used in LDM and other applications, a normalization mechanism called Group Normalization is often used to stabilize learning. In Group Normalization, a group of channels with a specified arbitrary constant is treated as one group, and normalization is performed within that group. Therefore, the number of channels of the image features input to Group Normalization must be a multiple of the specified arbitrary constant. In the above-described resizing method, the number of channels after resizing changes depending on the image size before and after resizing. Therefore, when resizing and combining the latent images or image features that are input to each layer of the first and second multilayer neural networks, it is difficult to make the number of channels of the combined tensor a multiple of an arbitrary constant. Furthermore, the number of channels of the resized condition image and the number of channels of the latent image or image feature to be combined with it may be an indivisible value, such as a prime number.

[0045] Therefore, the image processing unit 233 combines the latent image or image feature with the resized condition image, adjusts the number of channels using convolution processing, and then inputs the combined image to Group Normalization. In this way, by combining the images and then adjusting the number of channels through a convolution layer, the number of channels of the combined image of the condition images can be multiplied by an arbitrary constant. Therefore, the image processing unit 233 can reliably combine Group Normalization, a general mechanism for stabilizing learning, with the condition image.

[0046] Here, an example of using Group Normalization according to this embodiment will be described with reference to Fig. 4 and Fig. 5. Fig. 4 and Fig. 5 are diagrams showing an example of using Group Normalization according to this embodiment.

[0047] FIG. 4 shows an example of a Res Block that combines the above-mentioned resizing, convolution, and group normalization, and a first multilayer neural network encoder that uses the Res Block. Input corresponds to the latent image or image feature. Image corresponds to the condition image. Generally, the Res Block is widely used as a neural network mechanism. As shown in FIG. 4, the Res Block contains two Conv blocks, and two convolution operations are performed. Convolution operations generally combine activation functions such as SiLU (shown in Figure 4) with normalization such as Group Normalization. The presence of a convolution layer between the combination of image features and the resized condition image and the input to Group Normalization makes it possible to adjust the number of channels to any desired number. This allows the resizing described above to be incorporated into the Res Block mechanism shown in Figure 4.

[0048] By adjusting the number of channels using the convolutional layer after combining the image features and the resized condition image before Group Normalization, the resized condition image can be incorporated into the intermediate layer of the neural network. Therefore, in addition to the Res Block mechanism shown in Figure 4, this mechanism can also be applied to other commonly used Res Block mechanisms and Attention mechanisms when incorporating the resized condition image.

[0049] Figure 5 shows an example of an encoder for a first multilayer neural network incorporating the Res Block mechanism shown in Figure 4. Generally, in an LDM, a condition image is not input to the first multilayer neural network. However, as shown in Figure 5, by incorporating the Res Block mechanism shown in Figure 4, it becomes possible to input a condition image to the middle layer as well as the input layer of the first multilayer neural network.

[0050] Furthermore, when performing denoising and generation or conversion in the second multilayer neural network, the image processing unit 233 may use a semantic image having an ID for identifying an object on a pixel-by-pixel basis as auxiliary information. The image processing unit 233 uses the semantic image and the condition image together, resizes the semantic image, and combines it with the multi-resolution latent image or image feature. In image conversion, subject identification information is an important clue, and by combining a semantic image with a condition image and inputting it as a latent image or image feature, we can expect to improve the quality of image conversion. For example, in manga coloring, color consistency can be achieved across parts and frames of characters and objects. It also reduces coloring errors, such as mistakenly coloring skin with the color of clothing.

[0051] In addition to semantic images, general image information such as depth images, normal images, and posture estimation images, in which the subject corresponds to the source image on a pixel-by-pixel basis, can also be used as auxiliary information.

[0052] Furthermore, the image processing unit 233 may use a reference image as auxiliary information when performing denoising and generation or conversion in the second multilayer neural network. The image processing unit 233 uses both the reference image and the condition image, inputs the reference image to the encoder and converts it into image features, and then inputs the reference image together with the multi-resolution latent image or image features to the cross-attention mechanism. In this way, the quality of image conversion can be improved by using, as input, a reference image whose subject does not correspond pixel-by-pixel to the source image, in addition to the source image and auxiliary information whose subject corresponds pixel-by-pixel to the source image. For example, in coloring manga, color consistency can be achieved across parts and frames of characters and objects. Furthermore, the intended coloring of characters and objects can be more easily reflected. For example, incorrect coloring, such as the same character having different clothing or skin colors in different frames, can be reduced.

[0053] For example, in the case of image conversion in which the input is a manga line drawing and the output is a colored image, the reference image is a color image that can reference already-colored color information, showing the same subject on a different page from the line drawing. This reference image has a different composition from the source image, but contains subject information that is meaningful for image conversion. Since such a reference image does not correspond pixel-by-pixel to the subject in the source image, combining it with the condition image and inputting it is undesirable as it can actually result in a decrease in quality. Therefore, when using a reference image as input as auxiliary information for image conversion, the image processing unit 233 inputs the reference image itself or reference image features obtained from the reference image, for example, through an encoder, into the Query of the Cross Attention mechanism, and inputs the latent image or image features into the Key and Value.

[0054] Here, the functional configuration of the image processing unit 233 according to this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are diagrams showing an example of the functional configuration of the image processing unit 233 according to this embodiment. Fig. 6 shows a case where an auxiliary condition image is input as auxiliary information in combination with a condition image. Fig. 7 shows a case where a reference image is input as auxiliary information in combination with a condition image. The functional configuration of the image processing unit 233 shown in Figures 6 and 7 is also the functional configuration of the LDM. As shown in Figure 6, the image processing unit 233 includes an encoder 233a, a spreading unit 233b, a despreading unit 233c, and a decoder 233d.

[0055] The encoder 233a converts image data representing an input image x, which indicates a signal value for each pixel, into a latent representation z. The encoder 233a outputs the obtained latent representation to the diffusion unit 233b. The input image x is, for example, three-dimensional data indicating the signal value of each pixel distributed on a two-dimensional plane stretched in the horizontal and vertical directions for each color. For example, RGB image data is applied as the image data of the input image x. The latent representation is, for example, three-dimensional data that has been downsampled so that the number of elements in each of the horizontal and vertical directions is less than that of the original input image x. The number of elements in each of the horizontal and vertical directions of the latent representation z is, for example, 1 / 2 of the number of elements in each of the input image x. m m is an integer of 1 or greater.

[0056] The decoder 233d reconstructs an input image x' from the latent representation z. The amount of data of the reconstructed input image x' is equal to the amount of data of the original input image x input to the encoder 233a. That is, in the example of Figure 6, a latent representation with a smaller amount of data than the image data is processed in the diffusion process and de-diffusion process in the latent space. The encoder 233a and the decoder 233d function as an autoencoder in the image space. The autoencoder model is, for example, a variational encoder (VAE). The autoencoder model is trained in advance so that the difference between the original input image x and the reconstructed input image x' is minimized.

[0057] The diffusion unit 233b performs a diffusion process on the latent representation z input from the encoder 233a to generate the latent representation z t The noise component is generated as follows. The diffusion process is a process in which a noise addition step is repeatedly executed to generate noise-added data by adding a noise component to the data to be processed. t indicates the number of steps, i.e., the number of times the noise addition step is repeated. t can take a value between 0 and T depending on the noise intensity. The number of steps t is an integer equal to or greater than 2. The number of steps t is typically about 1000 (i.e., t is 0 to 1000, T=1000). The noise component is, for example, Gaussian noise. The latent expression z at step t t The mean value and variance of the distribution of sample values ​​constituting the noise component added at each step are set in advance so that the noise component can be obtained as follows:

[0058] The despreading unit 233c demultiplexes the latent representation z t A de-diffusion process is performed on the data to be processed to generate a latent representation z corresponding to the image data. The de-diffusion process is a process in which a noise removal step is repeatedly performed T times to remove noise components from the data to be processed and generate noise-removed data. The de-diffusion unit 233c includes T de-diffusion units 233c,t (t is an integer between 1 and T). Each of the T de-diffusion units 233c,t performs processing using a diffusion model. Each of the de-diffusion units 233c,t executes the noise removal step of step t. The latent representation z t is the data to be processed, and the obtained noise-removed data is used as the latent representation z t Here, the initial value of t is set to T, and the denoised data obtained for the first denoised data is set to the latent representation z. The diffusion model of the dediffusion unit 233c,t is the latent expression z t The despreading unit 233c,t converts the predicted noise component into the latent representation z t The noise-removed data is subtracted from the latent representation z t-1 is derived.

[0059] In the dediffusion process, a condition image is input to the input layer and hidden layer of the second multilayer neural network. The condition image includes at least the source image and may include an auxiliary condition image as auxiliary information. The diffusion model of the despreading units 233c,t includes multiple layers of computation blocks. Output data obtained from a layer is applied as input data to the next layer. The diffusion model of the despreading units 233c,t is, for example, U-Net or DiT (Stable Diffusion Models with Transformers). U-Net is a type of autoencoder and includes an encoder 233e and a decoder 233f, which are connected in series in that order. In each despreading unit 233c,t, a source image is input as a condition image to the input layer and the hidden layer, and an auxiliary condition image is input as auxiliary information. At this time, the source image and auxiliary information are resized to the same size as the latent image or image feature and combined.

[0060] When a reference image is used as auxiliary information, the reference image is input to the encoder 233g as shown in Fig. 7. The encoder 233g converts the input reference image into image features. After conversion, the encoder 233g inputs the image features obtained from the reference image to the input layer and hidden layer of the dediffusion units 233c,t, respectively.

[0061] (3-4) Output control unit 234 The output control unit 234 has a function of controlling the output of various information. For example, the output control unit 234 transmits the converted image generated and converted by the image processing unit 233 to the user terminal 10 via the communication unit 210 and displays it.

[0062] <3. Processing flow> The functional configuration of the image processing device 20 according to this embodiment has been described above. Next, the flow of processing according to this embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of the flow of processing according to this embodiment. It is assumed that the first multilayer neural network and the second multilayer neural network in the image processing device 20 have already been trained.

[0063] 8, first, the user selects a condition image including a source image to which image transformation is to be applied and auxiliary information, and a reference image (step S1). After the selection, the user transmits the selected condition image (source image and auxiliary information) and reference image to the image processing device 20 via the user terminal 10.

[0064] Next, the image processing unit 233 of the image processing device 20 resizes the condition image received by the communication unit 210 from the user terminal 10 to fit the second multilayer neural network (step S2). Here, the image processing unit 233 resizes the condition image to an image size suitable for the input layer and intermediate layer of the second multilayer neural network.

[0065] Next, the image processing unit 233 inputs each image to a second multilayer neural network (diffusion model) (step S3). Here, the image processing unit 233 combines the latent image or image feature created by random noise with the resized condition image, and inputs the combined image to the input layer and each intermediate layer of the second multilayer neural network. In this way, the image processing unit 233 performs denoising. The sampler used for denoising may be any sampler generally used for sampling diffusion models, such as DDIM or DPM Solver.

[0066] Next, the image processing unit 233 resizes the condition image received by the communication unit 210 from the user terminal 10 to fit the first multilayer neural network (VAE) (step S4). Here, the image processing unit 233 resizes the condition image to an image size suitable for the input layer and intermediate layer in the decoder of the first multilayer neural network.

[0067] Next, the image processing unit 233 inputs each image to a first multilayer neural network (VAE) (step S5). Here, the image processing unit 233 combines the latent image or image feature with the resized condition image, and inputs it to the input layer and each hidden layer of the first multilayer neural network. As a result, the image processing unit 233 obtains, as output, an RGB image, a grayscale image, or any other image in a general format that is meaningful on its own as a converted image.

[0068] The processing flow according to this embodiment has been described above. As described above, the image processing device 20 according to this embodiment includes an image processing unit 233 having a first multilayer neural network that converts between an RGB image and a Latent image, and a second multilayer neural network that performs a de-diffusion process on the Latent image. When decoding the Latent image that has undergone the de-diffusion process in the second multilayer neural network into an RGB image in the first multilayer neural network, the image processing unit 233 uses the Latent image or image features, and a condition image resized to the same size, as inputs to the input layer and intermediate layer of the first multilayer neural network.

[0069] With this configuration, when coloring an image with a small amount of information, such as a line drawing, the image processing device 20 can propagate the information contained in the input image to be colored to the output layer while minimizing information loss due to image conversion. Therefore, the image processing device 20 according to this embodiment makes it possible to improve the coloring accuracy in image conversion for coloring an image with a small amount of information, such as a line drawing.

[0070] <4. Modifications> The above describes the embodiments. Next, modifications of the above-described embodiments will be described. Note that each modification described below may be applied to the embodiments alone or in combination with each other. Furthermore, each modification may be applied in place of the configuration described in the embodiments, or may be applied in addition to the configuration described in the embodiments.

[0071] In the above-described embodiment, an example has been described in which the image processing system 1 includes the user terminal 10 and the image processing device 20 (server), but the present invention is not limited to such an example. The image processing system 1 may be realized by only the image processing device 20. The image processing device 20 may be realized by a device designed as a dedicated machine, or may be realized by a general computer such as a PC. In this case, the image processing device 20 is assumed to include storage such as a CPU, GPU, memory, and hard disk drive that would normally be included in a general computer. In addition, in the dedicated machine and the general computer, various processes are executed by a program that causes the system to function as the image processing device 20.

[0072] The above describes the modified examples. It should be noted that some or all of the functions of the image processing device 20 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing the functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, etc., and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs via networks such as the Internet or communication lines such as telephone lines, and devices that store programs for a certain period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in a computer system, or may be one that is realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

[0073] The embodiments of the present invention have been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope of the gist of the present invention. [Explanation of symbols]

[0074] 1...image processing system, 10...user terminal, 20...image processing device, 210...communication unit, 220...storage unit, 230...control unit, 231...learning data generation unit, 232...learning unit, 233...image processing unit, 233a...encoder, 233b...spreading unit, 233c...despreading unit, 233d...decoder, 233e...encoder, 233f...decoder, 233g...encoder, 234...output control unit, NW...network

Claims

1. an image processing unit including a first multilayer neural network for converting an RGB image and a Latent image mutually, and a second multilayer neural network for performing a de-diffusion process on the Latent image; Equipped with the image processing unit, when decoding the Latent image, on which the de-diffusion process has been performed in the second multilayer neural network, into the RGB image in the first multilayer neural network, uses the Latent image or image feature amounts and a condition image resized to the same size as the Latent image or image feature amounts as inputs to an input layer and an intermediate layer of the first multilayer neural network; Image processing device.

2. When the image processing unit performs the de-diffusion process in the second multilayer neural network, the image processing unit uses the latent image or the image feature quantity and a condition image resized to the same size as the latent image or the image feature quantity as inputs to an input layer and an intermediate layer of the second multilayer neural network. The image processing device according to claim 1 .

3. the image processing unit divides an image plane of the condition image into patches in accordance with the image size of the Latent image or the image feature of the multi-layer resolution processed in the multi-layer neural network, and resizes the condition image by rearranging and combining each pixel belonging to the patch in the channel axis direction.

3. The image processing device according to claim 1.

4. The image processing unit combines the latent image or the image feature amount with the resized condition image, adjusts the number of channels using convolution processing, and then inputs the combined image to Group Normalization.

3. The image processing device according to claim 1.

5. When performing denoising and generation or conversion in the second multilayer neural network, the image processing unit uses a semantic image having an ID that identifies an object in pixel units as auxiliary information, resizes the auxiliary information, and combines it with the Latent image or image feature of multilayer resolution.

3. The image processing device according to claim 1.

6. When performing denoising and generation or conversion in the second multilayer neural network, the image processing unit uses a reference image as auxiliary information, inputs the auxiliary information to an encoder and converts it into image features, and then inputs the image features to a Cross Attention mechanism together with the latent image of multilayer resolution or the image features.

3. The image processing device according to claim 1.

7. an image processing process including a first multi-layer neural network for converting an RGB image to a Latent image and vice versa, and a second multi-layer neural network for performing a de-diffusion process on the Latent image; Including, In the image processing step, when the Latent image, on which the de-diffusion process has been performed in the second multilayer neural network, is decoded into the RGB image in the first multilayer neural network, the Latent image or image feature quantities and a condition image resized to the same size are used together as inputs to an input layer and an intermediate layer of the first multilayer neural network. A computer-implemented image processing method.

8. Computer, an image processing means having a first multi-layer neural network for converting an RGB image to a Latent image and vice versa, and a second multi-layer neural network for performing a de-diffusion process on the Latent image; It functions as the image processing means, when decoding the Latent image, on which the de-diffusion process has been performed in the second multilayer neural network, into the RGB image in the first multilayer neural network, uses the Latent image or image feature quantities and a condition image resized to the same size as the Latent image or image feature quantities as inputs to an input layer and an intermediate layer of the first multilayer neural network; program.

Images