Learning device and inference device

Abstract

Provided are a learning device that efficiently generates image data while suppressing an operation amount when an image is generated using a machine learning model using a variational autoencoder (VAE), and an inference device that executes a predetermined inference process on target image data using the machine learning model.SOLUTION: An image processing device that exhibits a function as at least one of a learning device and an inference device includes a first machine learning model to which input image data is input from an image input unit, a second machine learning model to which a latent variable generated by the first machine learning model is input and from which output image data is output, and a learning process execution unit that executes a process of learning a setting value with the two machine learning models. The execution unit converts input image data into a latent variable using a first machine learning model, generates output image data from the latent variable using a second machine learning model to learn a setting value, maintains a brightness component of at least one of the input image data and the output image data, and downsamples only a tint component.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a learning device and an inference device that use a machine learning model for image processing. [Background technology]

[0002] In recent years, machine learning using neural networks has been applied to many fields. In particular, in fields such as image generation and image recognition, deep learning, which uses neural networks in a multi-layer structure, has demonstrated high recognition accuracy.

[0003] One example of this deep learning is the VAE (Variational Autoencoder, hereafter referred to as VAE), which converts input image data into a format known as a latent variable, thereby making it easier to calculate. This VAE adds a "probability distribution" to the "latent variable" used in the AE (Autoencoder) algorithm, and by changing the latent variable according to the probability distribution, it becomes possible to perform numerical analysis. As a result, it becomes easier to determine the exact state of the image and whether it matches the image data.

[0004] FIG. 9 shows an example of a case where input image data is compressed by an encoder 901 using a VAE, and the input image data is restored from the compressed data using a decoder 902. In this figure, the input image data is compressed by the encoder 901 in stages by downsampling to a smaller size, and then converted into latent variables in a latent space 903 by further reducing the number of channels or by performing vector quantization. In a VAE, even when the input image data is compressed into latent variables using the encoder 901, the features of the original image data are retained, so that the decoder 902 can restore the image data to a state close to the original. Note that in the latent space 903 where the latent variables are distributed, closer variables are located closer to each other, so the features of the output image can be easily adjusted by adjusting the latent variables.

[0005] An information processing device has been disclosed (see, for example, Patent Document 1) in which a machine learning model trains a neural network so that the probability distribution of feature information follows a normal distribution, making it possible to generate any exercise video from an exercise video including the user's body movements. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2023-183624 Summary of the Invention [Problem to be solved by the invention]

[0007] Image data used in machine learning models such as VAE is usually represented as a collection of pixels (picture elements) divided into coordinates, with each pixel assigned a pixel value (for example, 256 gradation values ​​of R (red), G (green), and B (blue) from 0 to 255). Then, all pixel values ​​of the target image data are input to input neurons to perform machine learning.

[0008] Incidentally, human vision is much more sensitive to changes in brightness than to changes in color, and humans have a visual characteristic that makes it difficult for them to perceive anything unnatural even when the color components are reduced. Ignoring this human visual characteristic and uniformly inputting / outputting pixel values ​​of input image data to the neurons of a machine learning model as described above is thought to reduce the efficiency of learning processing and increase the amount of calculations.

[0009] The same is true when using the VAE image generation model, where the input image data is compressed into a format known as a latent variable, converting the image data into a form that is easier to calculate.Here, the smaller the latent variable data volume and the more important features it captures, the more efficient the image generation will be.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a learning device and an inference device that can reduce the amount of calculations and generate image data more efficiently when generating images using a machine learning model that utilizes VAE. [Means for solving the problem]

[0011] In order to achieve the above-mentioned object, the present invention provides a learning device that learns setting values ​​in a machine learning model based on image data, comprising: an image input unit that accepts input image data; a first machine learning model to which the input image data is input from the image input unit; a second machine learning model to which latent variables generated by the first machine learning model are input and which outputs output image data; and a learning process execution unit that executes a process to learn setting values ​​in the first machine learning model and the second machine learning model, wherein the image input unit inputs the input image data to the first machine learning model, and the learning process execution unit learns setting values ​​using the first machine learning model to convert the input image data into latent variables, and using the second machine learning model to generate the output image data from the latent variables, and at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

[0012] In this learning device, the latent variables preferably follow an arbitrary probability distribution.

[0013] In this learning device, it is preferable that the image input unit inputs the brightness component of the input image data to an upper layer of the neural network of the first machine learning model, and inputs the downsampled color component to a lower layer than the upper layer.

[0014] In this learning device, it is preferable that the brightness component generated using the second machine learning model is output from an upper layer of the neural network of the second machine learning model, and the color component is output from a lower layer than the upper layer.

[0015] In addition, in order to achieve the above-mentioned object, the present invention provides an inference device that performs a predetermined inference process on image data, comprising an image input unit that accepts input of target image data, a first machine learning model to which the target image data is input from the image input unit, and an inference process execution unit that performs a predetermined inference process on the target image data, wherein the image input unit inputs the target image data to the first machine learning model, and the inference process execution unit converts the target image data into a latent variable using the first machine learning model learned by a learning device described in any one of claims 1 to 4.

[0016] It is preferable that this inference device further includes a third machine learning model for image processing and a text encoder for incorporating text input by a user into the third machine learning model, and that the latent variables output from the first machine learning model are processed using the third machine learning model and the text encoder.

[0017] In addition, in order to achieve the above-mentioned object, the present invention provides an inference device that performs a predetermined inference process on image data, comprising a second machine learning model into which latent variables are input, and an inference process execution unit that performs a predetermined inference process using the second machine learning model, wherein the inference process execution unit generates output image data from the latent variables using the second machine learning model learned by the learning device described above.

[0018] It is preferable that this inference device further includes a third machine learning model for image processing and a text encoder for incorporating user-input text into the third machine learning model, and that latent variables processed using the third machine learning model and the text encoder are input into the second machine learning model to generate the output image data.

[0019] In addition, in order to achieve the above-mentioned object, the present invention provides a machine learning model that converts image data features into latent variables, characterized in that setting values ​​are learned so as to convert input image data in which the brightness component is maintained and only the color component is downsampled into latent variables.

[0020] In addition, in order to achieve the above-mentioned object, the present invention is a machine learning model that generates image data from latent variables, characterized in that setting values ​​are learned from the latent variables so as to generate output image data in which the brightness component is maintained and only the color component is downsampled.

[0021] The present invention also provides a computer program that causes a computer to operate as the learning device or the inference device described above.

[0022] The present invention also provides a computer system that transmits the input data to the learning device or inference device or the computer program described above, and receives and uses the output data.

[0023] The present invention also provides a learning method for learning setting values ​​in a machine learning model based on image data, comprising an image input step for accepting input image data, and a learning process execution step for executing a process for learning setting values ​​in the machine learning model, wherein in the image input step, the input image data is input into a first machine learning model, and in the learning process execution step, setting values ​​are learned using the first machine learning model to convert the input image data into latent variables, and using a second machine learning model to generate output image data from the latent variables, and at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

[0024] The present invention also provides an inference method for performing a predetermined inference process on image data, comprising an image input step for accepting input of target image data, and an inference process execution step for performing a predetermined inference process on the target image data using a first machine learning model, wherein in the image input step, the target image data is input into the first machine learning model, and in the inference process execution step, the target image data is converted into a latent variable using the first machine learning model learned using the above-mentioned learning method. [Effects of the Invention]

[0025] A learning device according to the present invention includes an image input unit that accepts input image data, a first machine learning model to which the input image data is input from the image input unit, a second machine learning model to which latent variables generated by the first machine learning model are input and which outputs output image data, and a learning process execution unit that executes a process of learning setting values ​​for the first machine learning model and the second machine learning model. The image input unit inputs the input image data to the first machine learning model, and the learning process execution unit learns setting values ​​using the first machine learning model to convert the input image data into latent variables and using the second machine learning model to generate output image data from the latent variables, and at least one of the input image data and the output image data is image data in which only the color components are downsampled while the brightness component is maintained. With this configuration, the learning device according to the present invention can generate image data more efficiently with a reduced amount of calculation. [Brief explanation of the drawings]

[0026] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a functional block diagram of the image processing device. [Figure 3] FIG. 2 is a diagram showing a specific example of learning / inference processing in a machine learning model provided in the image processing device. [Figure 4]FIG. 2 is an image diagram of the learning / inference process in the first machine learning model provided in the image processing device. [Figure 5] FIG. 10 is an image diagram of the learning / inference process in the second machine learning model provided in the image processing device. [Figure 6] 10A to 10C are diagrams showing other examples of input image data used in the machine learning model. [Figure 7] FIG. 10 is a functional block diagram of an image processing device according to a first modified example of the embodiment of the present invention. [Figure 8] FIG. 10 is an explanatory diagram of image data in an image processing device according to a second modified example of the embodiment of the present invention. [Figure 9] This is an illustration of image compression and decompression when using a conventional VAE. DETAILED DESCRIPTION OF THE INVENTION

[0027] (Embodiment) An image processing device according to an embodiment of the present invention will be described with reference to Figures 1 to 8. In this embodiment, the image processing device functions as at least one of a learning device that learns setting values ​​in a machine learning model based on input image data for learning, and an inference device that executes a predetermined inference process on target image data using the machine learning model.

[0028] First, each processing unit included in the image processing device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the image processing device 1 includes a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and a reading unit 16. Note that, although the image processing device 1 and its operations will be described below as being implemented by a single server computer, the image processing device 1 may be configured to distribute processing among multiple computers.

[0029] The control unit 10 uses a processor such as a CPU and a memory to control the components of the device and realize various functions. The image processing unit 11 uses a processor such as a GPU or a dedicated circuit and a memory to execute image processing in accordance with control instructions from the control unit 10. The control unit 10 and the image processing unit 11 may be configured as a single piece of hardware (SoC: System on a Chip) that integrates a processor such as a CPU or a GPU, a memory, and further a storage unit 12 and a communication unit 13.

[0030] The storage unit 12 uses a hard disk or flash memory. The storage unit 12 stores an image processing program 1P and a machine learning library 1L that functions as a machine learning model (e.g., VAE). The storage unit 12 also stores definition data that defines the machine learning model, parameters including setting values ​​for the trained machine learning model, and the like.

[0031] The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet, etc. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

[0032] The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, etc. The display unit 14 is capable of displaying an image through processing by an image processing unit 11 in accordance with an instruction from the control unit 10.

[0033] The operation unit 15 includes a user interface such as a keyboard or a mouse. Physical buttons provided on the housing may be used, or software buttons displayed on the display unit 14 may be used. The operation unit 15 notifies the control unit 10 of operation information by the user.

[0034] The reading unit 16 can use, for example, a disk drive to read the image processing program 2P and the machine learning library 3L stored on a recording medium 2 that is an optical disk or the like. The image processing program 1P and the machine learning library 1L stored in the storage unit 12 may be the image processing program 2P and the machine learning library 3L that the reading unit 16 reads from the recording medium 2 and that the control unit 10 copies to the storage unit 12.

[0035] Next, the image processing functions of the image processing device 1 will be described with reference to Fig. 2. The control unit 10 of the image processing device 1 includes a learning process execution unit 101 and an inference process execution unit 102.

[0036] The learning process execution unit 101 functions as a machine learning model (machine learning engine) based on the machine learning library 1L, definition data, and parameter information stored in the storage unit 12. That is, the learning process execution unit 101 uses the machine learning model to be learned to execute a process of learning setting values ​​(parameters, etc.) in the machine learning model to be learned based on input image data for learning.

[0037] The inference processing execution unit 102 performs image processing based on the image processing program 1P stored in the storage unit 12. That is, the inference processing execution unit 102 uses a machine learning model to execute a predetermined inference process on input target image data.

[0038] The image input unit 110 accepts input of input image data for learning or target image data to be inferred. The learning process execution unit 101 outputs the input image data to a first machine learning model 111 via the image input unit 110. The inference process execution unit 102 outputs the target image data to the first machine learning model 111 via the image input unit 110.

[0039] In this case, the image input unit 110 converts the input image data and target image data into image data in which only the color components are downsampled while the brightness component remains the same, and inputs the converted image data to the first machine learning model 111. Specifically, in image conversion using a machine learning model using a VAE, the input image data is input as YCbCr rather than RGB, and the sampling rate of the CbCr components is lowered in advance while the Y component remains the same. This reduces the amount of input image data and allows brightness to be calculated with priority. Alternatively, the calculation speed can be increased by reducing the amount of data.

[0040] When the machine learning model is a VAE, the first machine learning model 111 corresponds to an encoder, and compresses the input image data into a small number of variables in a format called latent variables that follow an arbitrary probability distribution. More specifically, in the case of a VAE, the first machine learning model 111 obtains a mean vector μ and a variance vector σ from the features of the input image data, and probabilistically samples the latent variable z. In a VAE, the latent variables have a continuous distribution, so that the output features can be easily adjusted by adjusting the latent variables. Note that the arbitrary probability distribution may be a normal distribution (Gaussian distribution) or a uniform distribution based on vector quantization, and for example, different image data is mapped to different latent variable regions according to the normal distribution or uniform distribution.

[0041] On the other hand, the second machine learning model 112 corresponds to a decoder when the machine learning model is a VAE, and reconstructs (restores) the input image based on the latent variables extracted in the first machine learning model 111 during learning.

[0042] During learning, the machine learning models 111 and 112 can be trained by backpropagating the error between the input image and the output image and updating the parameters. In machine learning using a neural network, learning is performed by adjusting the weights, biases, etc. so that the error between the output and the correct answer is reduced. In this case, for example, backpropagation (error backpropagation method), which is an algorithm used during learning, updates the weights and biases by propagating the error back one layer at a time.

[0043] The image data restored by the second machine learning model 112 is input to the output unit 113. The output unit 113 can output the output image data output from the second machine learning model 112 to the storage unit 12. The output data may be rendered as an image in the image processing unit 11 and output to the display unit 14.

[0044] When the trained models are used, the first machine learning model 111 and the second machine learning model 112 can perform processes such as image generation and anomaly detection based on the trained parameters. Note that the first machine learning model 111 and the second machine learning model 112 may be part of a single machine learning model.

[0045] Next, a specific example of the learning / inference process in the machine learning models 111, 112 will be described with reference to Figures 3 to 5. As described above, the input image data 301 input from the image input unit 110 to the first machine learning model 111 is image data in which the brightness component is maintained and only the color component is downsampled. This input image data 301 is typically image data expressed using YCbCr rather than RBG.

[0046] Furthermore, in the description of this embodiment, the brightness component refers to a component related to the luminance, brightness, density, etc. of image data, and is typically the luminance component Y in YCbCr. In addition to Y in YCbCr, other components include L in Lab color space, V in HSV color space, and L in HLS color space. YUV or YPbPr may be used instead of YCbCr. Furthermore, color components typically refer to color difference components in YCbCr or hue and saturation in HSV. While RGB images and the like contain a mixture of brightness and color components, when used as image data for learning and inference, they may be converted to YCbCr or the like and then all or some of the components may be used. In this case, it is assumed that the image processing device 1 is equipped with a function and processing unit for color conversion from RGB images to YCbCr.

[0047] The first machine learning model 111 is a machine learning model that uses a neural network with a multi-layer structure, and typically, input image data 301 is stepwise downsampled along the layers of the neural network and compressed into latent variables. In this case, as shown in Figure 4, the Y data of the input image data 301 is input from the first layer (upper layer) of the first machine learning model 111 as usual, while the CbCr data with a reduced sample rate is input / output from the second layer (a lower layer than the upper layer). In other words, only the color components are input from the lower layers of the first machine learning model 111.

[0048] In the examples shown in FIGS. 3 and 4, when the size of the input image data 301 is 256*256 pixels, the Y data is input to the first layer at 256*256 pixels, whereas the CbCr data is input from the second layer, which is a separate route from the Y data, at a downsampled size of 128*128 pixels.

[0049] 5, the Y data generated using the second machine learning model 112 is output from an upper layer of the neural network of the second machine learning model 112, and the CbCr data is output from a lower layer. Note that when using the VAE of this method, the CbCr data is output in a downsampled state from the original image, so it may be used in combination with another model that upsamples the CbCr components.

[0050] As described above, in this embodiment, in the VAE conversion, input is performed in YCbCr rather than RGB, and the CbCr sample rate and bit depth are reduced in advance to reduce the amount of data. As a result, the amount of data of the input image data 301 input to the first machine learning model 111 is reduced in line with human visual characteristics, and lightness can be calculated with priority. Furthermore, because the Y data is not lost even when compressed into a latent variable, clear Y data can be restored from the latent variable. As a result, visually clear output image data 302 can be obtained that is in line with human visual characteristics.

[0051] In the above explanation, the downsampling rate is set to 1 / 2, but it may also be set to 1 / 4 or 1 / 8. In that case, the downsampled CbCr data is input from an even lower hierarchical level. By doing this, even if the input image is compressed to the same amount of data, the Y data is not lost, and when restored, the Y data is restored in greater detail, making it possible to obtain a visually clearer output image.

[0052] It is not necessary to maintain the brightness component and downsample only the color components of both the input image data 301 and the output image data 302. In other words, it is sufficient that at least one of the input image data 301 and the output image data 302 is image data that maintains the brightness component and downsamples only the color components.

[0053] Alternatively, the color components may be downsampled but retain their original size and used as the color components of the input image data. In this case, for example, as shown in Figure 6, downsampled YCbCr data as shown in (a) may be simply expanded to make the Y and CbCr components the same size as shown in (b), or the data may be further converted to RGB as shown in (c).

[0054] Next, a specific example of the learning / inference process of the first machine learning model 111 will be described. When the first machine learning model 111 is learning, the learning process execution unit 101 executes a process of learning setting values ​​in the first machine learning model 111 based on input image data 301 for learning. Note that when the first machine learning model 111 is learning, the image input unit 110 is set so that image data in which only the color components have been downsampled while the brightness component remains unchanged is input to the first machine learning model 111.

[0055] On the other hand, during inference by the first machine learning model 111, the image input unit 110 converts the target image data input to the first machine learning model 111 into image data in which only the color components are downsampled while the brightness component remains the same. The target image data is typically image data expressed using YCbCr. Then, during inference by the first machine learning model 111, latent variables are estimated from the target image data.

[0056] Next, a specific example of the learning / inference process of the second machine learning model 112 will be described. The latent variables generated by the first machine learning model 111 are input to the second machine learning model 112. When the second machine learning model 112 is learning, the learning process execution unit 101 learns the setting values ​​in the second machine learning model 112 so that input image data can be reconstructed based on the latent variables. At this time, the brightness component generated using the second machine learning model 112 is output from an upper layer, and the downsampled color components are output from a layer lower than the upper layer.

[0057] On the other hand, during inference by the second machine learning model 112, the latent variables input to the second machine learning model 112 are the latent variables generated by the first machine learning model 111. Then, the second machine learning model 112 operates to output image data in which only the color components have been downsampled while the brightness components remain unchanged, based on the latent variables.

[0058] As described above, the image processing device 1 according to the present embodiment is a learning device that learns setting values ​​in a machine learning model based on image data, and includes an image input unit 110 that accepts input image data 301, a first machine learning model 111 to which the input image data 301 is input from the image input unit 110, a second machine learning model 112 to which latent variables generated by the first machine learning model 111 are input and which outputs output image data 302, and a learning process execution unit 101 that executes a process of learning setting values ​​in the first machine learning model 111 and the second machine learning model 112. The image input unit 110 inputs the input image data 301 to the first machine learning model 111, and the learning process execution unit 101 learns setting values ​​to convert the input image data 301 into latent variables using the first machine learning model 111 and to generate output image data 302 from the latent variables using the second machine learning model 112, and at least one of the input image data 301 and the output image data 302 is image data in which the brightness component is maintained and only the color component is downsampled.

[0059] Traditionally, VAEs used in image generation models directly input and output image data in RGB format, so content loss (a loss function using recognition models such as VGG) is taken into account, but visual differences are not generally taken into account.

[0060] On the other hand, in the image processing device 1 according to the present embodiment, in image conversion using a machine learning model that utilizes VAE, input and output are performed using YCbCr instead of RGB, and the sample rate of the CbCr components is lowered in advance while leaving the Y component unchanged, thereby reducing the amount of data. As a result, it is possible to reduce the amount of input / output image data and calculate brightness with priority. Furthermore, since the amount of data of the input image to the encoder / the output image from the decoder is reduced, it is possible to increase the calculation speed.

[0061] In other words, in the image processing device 1 according to this embodiment, the latent variables have a smaller amount of data and accurately capture the brightness component, which is a more important feature of the input image data, thereby improving the efficiency of image generation in accordance with human visual characteristics.

[0062] Unlike conventional methods of calculating loss functions that prioritize brightness, the present invention reduces the amount of image data used in the machine learning model, thereby reducing the latent space in which the data resides.

[0063] (Variation 1) Next, a first modification of the image processing device 1 according to the embodiment of the present invention will be described with reference to Fig. 7. In the image processing device 1 according to the first modification, a third machine learning model (e.g., a diffusion model (U-Net)) 200 for image processing is prepared in the latent space between the first machine learning model 111 and the second machine learning model 112.

[0064] The image processing device 1 also includes a text encoder 201 for incorporating user-input text into the third machine learning model 200. At this time, for example, text information input by the user via a keyboard or the like using the operation unit 15 is input to the text encoder 201. The text encoder 201 converts text into vectors, and is, for example, a trained Transformer. The image processing device 1 then uses the third machine learning model 200 and the text encoder 201 to process the latent space output from the first machine learning model 111. Furthermore, the latent variables processed using the third machine learning model 200 and the text encoder 201 are input to the second machine learning model 112 to generate output image data.

[0065] It is well known that combining a VAE and a diffusion model (U-net) to generate an image that matches a text input. However, in this first modification, during learning, a latent space is constructed using input image data in which only the color components are downsampled while the brightness components remain unchanged. Then, a third machine learning model (i.e., a diffusion model) 200 is trained on this latent space to efficiently process and generate an image from text data. In this case, the processed and generated image data may be image data in which only the color components are downsampled while the brightness components remain unchanged.

[0066] Here, we will explain the learning / inference process of the third machine learning model 200 in this modification 1. The learning process execution unit 101 uses the third machine learning model 200 to learn setting values ​​based on the latent variables generated by the first machine learning model 111 and the input from the text encoder 201 so that input image data can be restored by passing it through the second machine learning model 112.

[0067] On the other hand, when a trained model is used (generation), the inference processing execution unit 102 inputs text information from the text encoder 201 to the third machine learning model 200, and finally passes the text information through the second machine learning model 112 to generate an image that matches the text information. For example, when a user inputs text (e.g., a dog dressed as Santa) using the operation unit 15, an image that matches the text (e.g., an image of a dog dressed as Santa) is efficiently processed and generated via the image processing unit 11.

[0068] (Variation 2) A second modification of the image processing device 1 according to the embodiment of the present invention will be described. In addition to the above-described method of downsampling only the color components of image data and then compressing them, an inverse pixel shuffler method may be used, which is a method of dividing an image into smaller multi-channel images.

[0069] That is, in this second modification, the brightness component Y and the downsampled color component CbCr data are each converted into images of the same size but with different numbers of channels, and then converted and inversely converted directly without downsampling. In this case, the CbCr data is not separately input from a lower hierarchical level of the first machine learning model 111, but is input and processed at the same hierarchical level as the Y data. Furthermore, the CbCr data is not separately output from a lower hierarchical level of the second machine learning model 112, but is output from the same hierarchical level as the Y data.

[0070] For example, as shown in Figure 8, when converting an image to 1 / 8, the Y data is divided into 64 channels, while the downsampled CbCr data has already been compressed to half its size, so it can be converted to 1 / 4, resulting in 16 + 16 channels. In other words, when the three-channel YCbCr data of the original image is reduced to 1 / 8 its size, conventionally it would be converted to 64 + 64 + 64 channel data, but in this modified example it is converted to 64 + 16 + 16 channel data. As a result, the latent variables have a smaller data volume, and the brightness component is captured more accurately than the color components, so that, as with the above embodiment, the calculation speed can be improved and the efficiency of image generation can be improved.

[0071] Of the hardware configuration of the image processing device 1 according to this embodiment, the communication unit 13, display unit 14, operation unit 15, and reading unit 16 are not essential. The communication unit 13 may be used once, for example, when acquiring the image processing program 1P and machine learning library 1L stored in the storage unit 12 from an external server device, and then may not be used again. Similarly, the reading unit 16 may not be used after reading and acquiring the image processing program 1P and machine learning library 1L from a storage medium. The communication unit 13 and reading unit 16 may be the same device using serial communication such as USB.

[0072] The image processing device 1 may be configured as a web server to provide the functions of the first machine learning model 111 and the second machine learning model 112 to a web client device that includes a display unit and a communication unit. In this case, the communication unit 13 is used to receive requests from the web client device and transmit processing results.

[0073] The error used during learning may be an appropriate function such as square error, absolute value error, or cross entropy error depending on the input / output data and the learning purpose.

[0074] The present invention is not limited to the configurations of the above-described embodiments, and various modifications are possible within the scope of the invention. Furthermore, to achieve the object of the present invention, the present invention can be realized as an image processing method (learning method and inference method) having characteristic constituent means included in an image processing device (learning device and inference device) as steps, or as a program including those characteristic steps. The program can be stored in a ROM or the like, or can be distributed via a recording medium such as a USB memory or a communication network.

[0075] The present invention can also be realized as a computer system that transmits input data to an image processing device or a computer program and receives and uses output data from the image processing device or the computer program. This system is a processing system that uses data obtained from a machine learning model that has been trained through the above-described processing, and can provide a variety of services. The device used in this system is an image processing device equipped with a display unit and a communication unit, or an information processing device that can send and receive information to and from a computer, such as a so-called PC, smartphone, mobile terminal, or game console. [Explanation of symbols]

[0076] 1 Image processing device (learning device and inference device) 12 Memory unit (learning result memory unit) 101 Learning processing execution unit 102 Inference processing execution unit 110 Image input unit 111 First Machine Learning Model 112 Second Machine Learning Model 113 Output section 200 Third Machine Learning Model 201 Text Encoder 301 Input image data 302 Output image data

Claims

1. A learning device that learns setting values ​​in a machine learning model based on image data, an image input unit that accepts input image data; a first machine learning model to which the input image data is input from the image input unit; a second machine learning model to which the latent variables generated by the first machine learning model are input and which outputs output image data; a learning process execution unit that executes a process of learning setting values ​​in the first machine learning model and the second machine learning model, the image input unit inputs the input image data to the first machine learning model; the learning process execution unit learns setting values ​​to convert the input image data into latent variables using the first machine learning model, and to generate the output image data from the latent variables using the second machine learning model; A learning device characterized in that at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

2. 2. The learning device according to claim 1, wherein the latent variables follow an arbitrary probability distribution.

3. The learning device according to claim 1, characterized in that the image input unit inputs the brightness component of the input image data to an upper layer of the neural network of the first machine learning model, and inputs the downsampled color component to a lower layer than the upper layer.

4. The learning device according to claim 1, characterized in that the brightness component generated using the second machine learning model is output from an upper layer of the neural network of the second machine learning model, and the color component is output from a layer lower than the upper layer.

5. An inference device that performs a predetermined inference process on image data, an image input unit that accepts input of target image data; a first machine learning model to which the target image data is input from the image input unit; an inference processing execution unit that executes a predetermined inference processing on the target image data, the image input unit inputs the target image data into the first machine learning model; The inference processing execution unit converts the target image data into latent variables using a first machine learning model trained by the learning device described in any one of claims 1 to 4.

6. The inference device further comprises: A third machine learning model for image processing; a text encoder for incorporating user-inputted text into the third machine learning model; 6. The inference device according to claim 5, wherein the latent variables output from the first machine learning model are processed using the third machine learning model and the text encoder.

7. An inference device that performs a predetermined inference process on image data, a second machine learning model into which the latent variables are input; an inference processing execution unit that executes a predetermined inference processing using the second machine learning model, The inference processing execution unit generates output image data from latent variables using a second machine learning model trained by the learning device described in any one of claims 1 to 4.

8. The inference device further comprises: A third machine learning model for image processing; a text encoder for incorporating user-inputted text into the third machine learning model; The inference device according to claim 7, characterized in that the latent variables processed using the third machine learning model and the text encoder are input into the second machine learning model to generate the output image data.

9. A machine learning model that converts features of image data into latent variables, A machine learning model characterized by having settings learned to convert input image data in which the brightness component is maintained and only the color component is downsampled into a latent variable.

10. A machine learning model that generates image data from latent variables, A machine learning model characterized in that settings are learned from latent variables to generate output image data in which the brightness component is maintained and only the color component is downsampled.

11. A computer program causing a computer to operate as the learning device according to any one of claims 1 to 4.

12. A computer program that causes a computer to operate as the inference device according to any one of claims 5 to 8.

13. A learning method for learning setting values ​​in a machine learning model based on image data, comprising: an image input step of accepting input image data; a learning process execution step of executing a process of learning a setting value in a machine learning model, In the image input step, the input image data is input to a first machine learning model; In the learning process execution step, a setting value is learned so as to convert the input image data into a latent variable using the first machine learning model, and so as to generate output image data from the latent variable using a second machine learning model; A learning method characterized in that at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

14. An inference method for performing a predetermined inference process on image data, an image input step of accepting input of target image data; an inference processing execution step of executing a predetermined inference processing on the target image data using the first machine learning model; In the image input step, the target image data is input to the first machine learning model; An inference method characterized in that, in the inference processing execution step, the target image data is converted into latent variables using the first machine learning model learned by the learning method described in claim 13.

Images