Learning and reasoning devices

Abstract

Provided is a learning device that reduces the amount of calculations and efficiently generates image data when generating images using a machine learning model that utilizes a variational autoencoder (VAE), and an inference device that executes a specified inference process on target image data using the machine learning model. [Solution] An image processing device that functions as at least one of a learning device and an inference device includes a first machine learning model to which input image data is input from an image input unit, a second machine learning model to which latent variables generated by the machine learning model are input and output image data is output, and a learning process execution unit that executes a process of learning setting values ​​with the two machine learning models. The execution unit converts the input image data into latent variables using the first machine learning model, and generates output image data from the latent variables using the second machine learning model to learn setting values, maintains the brightness component of at least one of the input image data and the output image data, and downsamples only the color components.

Description

[Technical field]

[0001] The present invention relates to a learning device and an inference device that use a machine learning model for image processing. [Background technology]

[0002] In recent years, machine learning using neural networks has been applied to many fields. In particular, deep learning, which uses neural networks in a multi-layer structure, has demonstrated high recognition accuracy in fields such as image generation and image recognition.

[0003] Among the deep learning techniques, for example, VAE (Variational Autoencoder, hereafter referred to as VAE) is widely used, which converts input image data into a format called latent variables to make the image data easier to calculate. This VAE adds a "probability distribution" to the "latent variable" portion used in the AE (Autoencoder) algorithm, and by changing the latent variable according to the probability distribution, it becomes possible to perform numerical analysis. As a result, it becomes easier to determine the exact state of the image and whether it matches the image data.

[0004] FIG. 9 shows an example of compressing input image data by an encoder 901 using a VAE, and restoring the input image data from the compressed data by a decoder 902. In this figure, the input image data is compressed on the encoder 901 side by stepwise downsampling to a smaller size, and then converted into latent variables in a latent space 903 by further reducing the number of channels or by performing vector quantization. In a VAE, even if the input image data is compressed into latent variables using the encoder 901, the characteristics of the original image data are maintained, so that the image data can be restored to a state close to the original image data on the decoder 902 side. Note that in the latent space 903 where the latent variables are distributed, closer variables are located closer to each other, so the characteristics of the output image can be easily adjusted by adjusting the latent variables.

[0005] An information processing device has been disclosed in which a machine learning model trains a neural network so that the probability distribution of feature information follows a normal distribution, making it possible to generate any exercise video from an exercise video including a user's body movements (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2023-183624 A Summary of the Invention [Problem to be solved by the invention]

[0007] Typically, image data used in machine learning models such as VAE is represented as a collection of pixels (picture elements) partitioned by coordinates, and each pixel is assigned a pixel value (for example, 256 gradation values ​​of R (red), G (green), and B (blue) from 0 to 255). Then, all pixel values ​​of the target image data are input to input neurons to execute machine learning.

[0008] However, human vision is much more sensitive to changes in brightness than to changes in color, and humans have visual characteristics that make them less likely to perceive something as unnatural even if the color components are reduced. By ignoring such human visual characteristics and uniformly inputting / outputting the pixel values ​​of the input image data to the neurons of the machine learning model as described above, it is believed that the learning processing efficiency is reduced and the amount of calculations is increased.

[0009] The same is true when using the VAE image generation model, where the input image data is compressed into a format known as a latent variable, converting the image data into a form that is easier to calculate. Here, the less data the latent variables have and the more important features they capture, the more efficient the image generation will be.

[0010] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a learning device and an inference device that can reduce the amount of calculations and generate image data more efficiently when generating images using a machine learning model that utilizes VAE. [Means for solving the problem]

[0011] In order to achieve the above object, the present invention provides a learning device that learns setting values ​​in a machine learning model based on image data, the learning device including: an image input unit that receives input image data and converts the input image data into image data in which only the color components are downsampled while the brightness components are left as is; Converted The system comprises a first machine learning model to which the input image data is input and which generates compressed latent variables, a second machine learning model to which the latent variables generated by the first machine learning model are input and which outputs output image data, and a learning process execution unit that executes a process of learning setting values ​​in the first machine learning model and the second machine learning model, wherein the image input unit inputs the input image data to the first machine learning model, and the learning process execution unit learns setting values ​​to convert the input image data into the latent variables using the first machine learning model, and to generate the output image data that has restored the input image data from the latent variables using the second machine learning model, and all image data of at least one of the input image data and the output image data is image data in which the brightness components are maintained and only the color components are downsampled.

[0012] In this learning device, it is preferable that the latent variables follow an arbitrary probability distribution.

[0013] In this learning device, it is preferable that the image input unit inputs the brightness component of the input image data to an upper layer of the neural network of the first machine learning model, and inputs the downsampled color component to a lower layer than the upper layer.

[0014] In this learning device, it is preferable that the brightness component generated using the second machine learning model is output from an upper layer of the neural network of the second machine learning model, and the color component is output from a lower layer than the upper layer.

[0015] In addition, in order to achieve the above-mentioned object, the present invention provides an inference device that performs a predetermined inference process on image data, comprising an image input unit that accepts input of target image data, a first machine learning model to which the target image data is input from the image input unit, and an inference process execution unit that performs a predetermined inference process on the target image data, wherein the image input unit inputs the target image data to the first machine learning model, and the inference process execution unit converts the target image data into a latent variable using the first machine learning model learned by a learning device described in any one of claims 1 to 4.

[0016] In this inference device, it is preferable that the inference device further includes a third machine learning model for image processing and a text encoder for incorporating text input by a user into the third machine learning model, and that the third machine learning model and the text encoder are used to process the latent variables output from the first machine learning model.

[0017] In addition, in order to achieve the above-mentioned object, the present invention provides an inference device that executes a predetermined inference process on image data, comprising a second machine learning model into which latent variables are input, and an inference process execution unit that executes a predetermined inference process using the second machine learning model, wherein the inference process execution unit generates output image data from the latent variables using the second machine learning model learned by the learning device described above.

[0018] In this inference device, it is preferable that the inference device further includes a third machine learning model for image processing and a text encoder for incorporating text input by a user into the third machine learning model, and that the latent variables processed using the third machine learning model and the text encoder are input into the second machine learning model to generate the output image data.

[0021] The present invention also provides a computer program for causing a computer to operate as the above-described learning device or inference device.

[0022] The present invention also provides a computer system that transmits the input data to the above-described learning device or inference device or the above-described computer program, and receives and utilizes the output data.

[0023] The present invention also provides a learning method executed by a learning device that learns setting values ​​in a machine learning model based on image data, the method including: receiving input image data; hair, Converts the image data into one where the brightness component remains the same and only the color component is downsampled. Image Input and a learning process execution step of executing a process of learning a setting value in a machine learning model, wherein in the image input step, Converted The input image data is input into a first machine learning model, and in the learning process execution step, setting values ​​are learned to convert the input image data into latent variables using the first machine learning model, and to generate output image data that restores the input image data from the latent variables using a second machine learning model, and all image data of at least one of the input image data and the output image data is image data in which the brightness components are maintained and only the color components are downsampled.

[0024] The present invention also provides a method for executing a predetermined inference process on image data. The inference device executesAn inference method comprising: an image input step of accepting input of target image data; and an inference process execution step of executing a predetermined inference process on the target image data using a first machine learning model, wherein in the image input step, the target image data is input to the first machine learning model; and in the inference process execution step, the target image data is converted into a latent variable using the first machine learning model learned by the above learning method. Effect of the Invention

[0025] The learning device according to the present invention includes an image input unit that accepts input image data, a first machine learning model to which the input image data is input from the image input unit, a second machine learning model to which latent variables generated by the first machine learning model are input and output image data is output, and a learning process execution unit that executes a process of learning setting values ​​in the first machine learning model and the second machine learning model. The image input unit inputs the input image data to the first machine learning model, and the learning process execution unit learns setting values ​​to convert the input image data into latent variables using the first machine learning model and to generate output image data from the latent variables using the second machine learning model, and at least one of the input image data and the output image data is image data in which only the color components are downsampled while the brightness components are maintained. With this configuration, the learning device according to the present invention can reduce the amount of calculation and generate image data more efficiently. [Brief description of the drawings]

[0026] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a functional block diagram of the image processing device. [Diagram 3] FIG. 2 is a diagram showing a specific example of learning / inference processing in a machine learning model provided in the image processing device. [Figure 4] FIG. 2 is an image diagram of the learning / inference process in the first machine learning model provided in the image processing device. [Diagram 5]FIG. 2 is an image diagram of the learning / inference process in the second machine learning model provided in the image processing device. [Figure 6] 13A to 13C are diagrams showing other examples of input image data used in the above machine learning model. [Figure 7] FIG. 2 is a functional block diagram of an image processing device according to a first modified example of an embodiment of the present invention. [Figure 8] FIG. 11 is an explanatory diagram of image data in an image processing device according to a second modified example of the embodiment of the present invention. [Figure 9] This is an illustration of image compression and decompression when using a conventional VAE. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] (Embodiment) An image processing device according to an embodiment of the present invention will be described with reference to Figures 1 to 8. In this embodiment, the image processing device functions as at least one of a learning device that learns setting values ​​in a machine learning model based on input image data for learning, and an inference device that executes a predetermined inference process on target image data using the machine learning model.

[0028] First, each processing unit of the image processing device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the image processing device 1 includes a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and a reading unit 16. Note that, although the image processing device 1 and the operation in the image processing device 1 will be described below as a single server computer, the image processing device 1 may be configured to distribute processing among multiple computers.

[0029] The control unit 10 uses a processor such as a CPU and a memory to control the components of the device to realize various functions. The image processing unit 11 uses a processor such as a GPU or a dedicated circuit and a memory to execute image processing in response to control instructions from the control unit 10. The control unit 10 and the image processing unit 11 may be configured as a single piece of hardware (SoC: System on a Chip) that integrates processors such as a CPU and a GPU, memory, and further a storage unit 12 and a communication unit 13.

[0030] The storage unit 12 uses a hard disk or a flash memory. The storage unit 12 stores an image processing program 1P and a machine learning library 1L that functions as a machine learning model (for example, VAE). The storage unit 12 also stores definition data that defines the machine learning model, parameters including setting values ​​in the trained machine learning model, and the like.

[0031] The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet, etc. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

[0032] The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, etc. The display unit 14 is capable of displaying an image by processing in the image processing unit 11 according to an instruction from the control unit 10.

[0033] The operation unit 15 includes a user interface such as a keyboard or a mouse. A physical button provided on the housing may be used, or a software button or the like displayed on the display unit 14 may be used. The operation unit 15 notifies the control unit 10 of operation information by the user.

[0034] The reading unit 16 can use, for example, a disk drive to read the image processing program 2P and the machine learning library 3L stored in the recording medium 2, which is an optical disk or the like. The image processing program 1P and the machine learning library 1L stored in the storage unit 12 may be the image processing program 2P and the machine learning library 3L read by the reading unit 16 from the recording medium 2 and copied into the storage unit 12 by the control unit 10.

[0035] Next, the image processing function of the image processing device 1 will be described with reference to Fig. 2. The control unit 10 of the image processing device 1 includes a learning process execution unit 101 and an inference process execution unit 102.

[0036] The learning process execution unit 101 functions as a machine learning model (machine learning engine) based on the machine learning library 1L, definition data, and parameter information stored in the storage unit 12. That is, the learning process execution unit 101 executes a process of learning setting values ​​(parameters, etc.) in the machine learning model to be learned based on input image data for learning, using the machine learning model to be learned.

[0037] The inference processing execution unit 102 performs image processing based on the image processing program 1P stored in the storage unit 12. That is, the inference processing execution unit 102 uses a machine learning model to execute a predetermined inference processing on the input target image data.

[0038] The image input unit 110 accepts input of input image data for learning or target image data to be inferred. The learning process execution unit 101 outputs the input image data to a first machine learning model 111 via the image input unit 110. The inference process execution unit 102 outputs the target image data to the first machine learning model 111 via the image input unit 110.

[0039] At this time, the image input unit 110 converts the input image data and the target image data into image data in which only the color components are downsampled while the brightness components remain the same, and inputs the converted image data to the first machine learning model 111. Specifically, in the image conversion of the machine learning model using the VAE, the input image data is input in YCbCr rather than RGB, and the sample rate of the CbCr components is lowered in advance while the Y component remains the same. This reduces the amount of data of the input image data, and allows the brightness to be calculated with priority. Alternatively, the calculation speed can be increased by reducing the amount of data.

[0040] When the machine learning model is a VAE, the first machine learning model 111 corresponds to an encoder, and compresses the input image data into a small number of variables in a format called latent variables that follow an arbitrary probability distribution. More specifically, in the case of a VAE, the first machine learning model 111 obtains a mean vector μ and a variance vector σ from the features of the input image data, and probabilistically samples the latent variables z. In a VAE, since the latent variables have a continuous distribution, the output features can be easily adjusted by adjusting the latent variables. Note that the arbitrary probability distribution can be a normal distribution (Gaussian distribution) or a uniform distribution based on vector quantization, and for example, different image data is mapped to different latent variable regions according to a normal distribution or a uniform distribution.

[0041] On the other hand, the second machine learning model 112 corresponds to a decoder when the machine learning model is a VAE, and reconstructs (restores) the input image based on the latent variables extracted in the first machine learning model 111 during learning.

[0042] During learning, the machine learning models 111 and 112 can be trained by updating parameters through backpropagation of the error between the input image and the output image. In machine learning using a neural network, learning is performed by adjusting weights, biases, etc. so that the error between the output and the correct answer is reduced. In this case, for example, in backpropagation (error backpropagation method), which is an algorithm used during learning, the weights and biases are updated by propagating the error back one layer at a time.

[0043] The image data restored by the second machine learning model 112 is input to the output unit 113. The output unit 113 can output the output image data output from the second machine learning model 112 to the storage unit 12. Note that the output data may be rendered as an image in the image processing unit 11 and output to the display unit 14.

[0044] When the trained models are used, the first machine learning model 111 and the second machine learning model 112 can perform processes such as image generation and anomaly detection based on already trained parameters. Note that the first machine learning model 111 and the second machine learning model 112 may be part of one machine learning model.

[0045] Next, a specific example of the learning / inference process in the machine learning models 111, 112 will be described with reference to Fig. 3 to Fig. 5. As described above, the input image data 301 input from the image input unit 110 to the first machine learning model 111 is image data in which the brightness component is maintained and only the color components are downsampled. This input image data 301 is typically image data expressed using YCbCr rather than RBG.

[0046] In addition, the brightness component in the description of this embodiment is a component related to the luminance, brightness, density, etc. of image data, and is typically the luminance component Y in YCbCr. In addition to Y in YCbCr, there is L in Lab color space, V in HSV color space, L in HLS color space, etc. YUV or YPbPr may be used instead of YCbCr. Furthermore, the color component is typically a color difference component in YCbCr or a hue and saturation in HSV. In addition, although an RGB image or the like is a mixture of brightness components and color components, when it is used as image data for learning and inference, it is possible to convert it to YCbCr or the like and then use all or a part of the components. In this case, it is assumed that the image processing device 1 is provided with a function and a processing unit for color conversion from an RGB image to YCbCr.

[0047] The first machine learning model 111 is a machine learning model using a neural network with a multi-layer structure, and typically, the input image data 301 is stepwise downsampled and compressed into latent variables along the hierarchical layers of the neural network. At this time, as shown in Fig. 4, the Y data of the input image data 301 is input from the first hierarchical layer (upper hierarchical layer) in the same manner as in the normal case of the first machine learning model 111, while the CbCr data with a reduced sample rate is input / output from the second hierarchical layer (a lower hierarchical layer than the upper hierarchical layer). In other words, only the color components are input from the lower hierarchical layers of the first machine learning model 111.

[0048] In the example shown in FIGS. 3 and 4, when the size of the input image data 301 is 256*256 pixels, the Y data is input to the first layer at 256*256 pixels, whereas the CbCr data is downsampled to a size of 128*128 pixels and input from the second layer, which is a different route from the Y data.

[0049] 5, the Y data generated using the second machine learning model 112 is output from an upper layer of the neural network of the second machine learning model 112, and the CbCr data is output from a lower layer. When the VAE of this method is used, the CbCr data is output in a downsampled state from the original image, so it may be used in combination with another model that upsamples the CbCr components.

[0050] As described above, in the present embodiment, in the VAE conversion, input is performed in YCbCr instead of RGB, and the sample rate and bit depth of CbCr are reduced in advance to reduce the amount of data. As a result, the amount of data of the input image data 301 input to the first machine learning model 111 is reduced in accordance with human visual characteristics, and lightness can be calculated with priority. Furthermore, since the Y data is not lost even when compressed into a latent variable, clear Y data can be restored from the latent variable, and as a result, visually clear output image data 302 in accordance with human visual characteristics can be obtained.

[0051] In the above explanation, the downsampling rate is set to 1 / 2, but it may be set to 1 / 4 or 1 / 8. In that case, the downsampled CbCr data is input from an even lower hierarchical level. By doing so, even if the input image is compressed to the same amount of data, the Y data is not lost, and at the same time, the Y data is restored in more detail at the time of restoration, making it possible to obtain a visually clearer output image.

[0052] It is not necessary that both the input image data 301 and the output image data 302 are image data in which the brightness component is maintained and only the color components are downsampled. In other words, it is sufficient that at least one of the input image data 301 and the output image data 302 is image data in which the brightness component is maintained and only the color components are downsampled.

[0053] Also, the color components may be downsampled but the size may be maintained as the color components of the input image data. In this case, for example, as shown in Fig. 6, from the downsampled YCbCr data as shown in (a), the color components may be simply expanded as shown in (b) to make the Y component and CbCr component the same size, or the data may be further converted to RGB as shown in (c) and used.

[0054] Next, a specific example of the learning / inference process of the first machine learning model 111 will be described. When the first machine learning model 111 is learning, the learning process execution unit 101 executes a process of learning setting values ​​in the first machine learning model 111 based on input image data 301 for learning. Note that when the first machine learning model 111 is learning, the image input unit 110 is set so that image data in which only the color components are downsampled while the brightness components are left as they are is input to the first machine learning model 111.

[0055] On the other hand, during inference in the first machine learning model 111, the image input unit 110 converts the target image data input to the first machine learning model 111 into image data in which only the color components are downsampled while the brightness components are left as is. The target image data is typically image data expressed using YCbCr. During inference in the first machine learning model 111, latent variables are estimated from the target image data.

[0056] Next, a specific example of the learning / inference process of the second machine learning model 112 will be described. The latent variables generated by the first machine learning model 111 are input to the second machine learning model 112. During learning of the second machine learning model 112, the learning process execution unit 101 learns setting values ​​in the second machine learning model 112 so that input image data can be reconstructed based on the latent variables. At this time, the brightness component generated using the second machine learning model 112 is output from an upper layer, and the downsampled color components are output from a layer lower than the upper layer.

[0057] On the other hand, during inference by the second machine learning model 112, the latent variables input to the second machine learning model 112 are the latent variables generated in the first machine learning model 111. Then, the second machine learning model 112 operates to output image data in which only the color components have been downsampled while the brightness components remain unchanged, based on the latent variables.

[0058] As described above, the image processing device 1 according to the present embodiment is a learning device that learns setting values ​​in a machine learning model based on image data, and includes an image input unit 110 that accepts input image data 301, a first machine learning model 111 to which the input image data 301 is input from the image input unit 110, a second machine learning model 112 to which a latent variable generated by the first machine learning model 111 is input and output image data 302, and a learning process execution unit 101 that executes a process of learning setting values ​​in the first machine learning model 111 and the second machine learning model 112. The image input unit 110 inputs the input image data 301 to the first machine learning model 111, and the learning process execution unit 101 learns setting values ​​to convert the input image data 301 into a latent variable using the first machine learning model 111 and to generate output image data 302 from the latent variable using the second machine learning model 112, and at least one of the input image data 301 and the output image data 302 is image data in which the brightness component is maintained and only the color component is downsampled.

[0059] Conventionally, VAE used in image generation models directly inputs and outputs image data in RGB format, so content loss (a loss function using recognition models such as VGG) is taken into account, but visual differences are not generally taken into account.

[0060] On the other hand, in the image processing device 1 according to the present embodiment, in image conversion using a machine learning model that utilizes VAE, input and output are performed in YCbCr rather than RGB, and the sample rate of the CbCr component is lowered in advance while leaving the Y component unchanged, thereby reducing the amount of data. As a result, it is possible to reduce the data of the input / output image data and to calculate the brightness with priority. In addition, since the amount of data of the input image to the encoder / the output image from the decoder is reduced, the calculation speed can be increased.

[0061] In other words, in the image processing device 1 according to the present embodiment, the latent variables have a smaller amount of data and accurately capture the brightness component, which is a more important feature of the input image data, so that the efficiency of image generation can be improved in accordance with the human visual characteristics.

[0062] Unlike conventional loss functions that prioritize brightness in their calculations, the present invention reduces the amount of image data used in the machine learning model, thereby reducing the latent space in which the data exists.

[0063] (Variation 1) Next, a first modified example of the image processing device 1 according to the embodiment of the present invention will be described with reference to Fig. 7. In the image processing device 1 according to the first modified example, a third machine learning model (e.g., a diffusion model (U-Net)) 200 for image processing is prepared in a latent space between the first machine learning model 111 and the second machine learning model 112.

[0064] The image processing device 1 also includes a text encoder 201 for incorporating user-input text from the side of the third machine learning model 200. At this time, for example, text information input by the user via a keyboard or the like using the operation unit 15 is input to the text encoder 201. The text encoder 201 converts text into a vector, and is, for example, a trained Transformer. Then, the image processing device 1 uses the third machine learning model 200 and the text encoder 201 to process the latent space output from the first machine learning model 111. Furthermore, the latent variables processed using the third machine learning model 200 and the text encoder 201 are input to the second machine learning model 112 to generate output image data.

[0065] It is a well-known technique to combine a VAE and a diffusion model (U-net) to generate an image according to a text input. However, in this modified example 1, during learning, a latent space is constructed using input image data in which only the color components are downsampled while the brightness components remain unchanged. Then, a third machine learning model (i.e., a diffusion model) 200 is trained on this latent space to efficiently process and generate an image from text data. At this time, the image data to be processed and generated may be image data in which only the color components are downsampled while the brightness components remain unchanged.

[0066] Here, a description will be given of the learning / inference process of the third machine learning model 200 in this modified example 1. The learning process execution unit 101 uses the third machine learning model 200 to learn setting values ​​based on the latent variables generated by the first machine learning model 111 and the input from the text encoder 201, so that the input image data can be restored by passing it through the second machine learning model 112.

[0067] On the other hand, when using a trained model (generation), the inference process execution unit 102 inputs text information from the text encoder 201 to the third machine learning model 200, and finally passes the text information through the second machine learning model 112 to generate an image according to the text information. For example, when a user inputs text (e.g., a dog dressed as Santa) using the operation unit 15, an image according to the input (e.g., an image of a dog dressed as Santa) is efficiently processed and generated via the image processing unit 11.

[0068] (Variation 2) A second modified example of the image processing device 1 according to the embodiment of the present invention will be described. In addition to the above-mentioned method of downsampling only the color components of image data and then compressing them, an inverse pixel shuffler method of dividing an image into small multi-channel images may be used.

[0069] That is, in the present modification 2, the brightness component Y and the downsampled color component CbCr data are converted into images of the same size but with different numbers of channels, and then converted and inversely converted as they are without downsampling. In this case, the CbCr data is not separately input from a lower hierarchical level of the first machine learning model 111, but is input and processed at the same hierarchical level as the Y data. Also, the CbCr data is not separately output from a lower hierarchical level of the second machine learning model 112, but is output from the same hierarchical level as the Y data.

[0070] For example, as shown in Fig. 8, when converting an image to 1 / 8, Y data is divided into 64 channels, whereas the downsampled CbCr data has already been compressed to half, so it is sufficient to convert it to 1 / 4 and have 16+16 channels. In other words, when the three-channel data of the YCbCr data of the original image is reduced to 1 / 8 size, it would have been converted to 64+64+64 channel data in the past, but in this modified example, it is converted to 64+16+16 channel data. Therefore, the amount of data for the latent variables is reduced, and the brightness component is captured more accurately than the color components, so that the calculation speed can be improved and the efficiency of image generation can be improved, as in the above embodiment.

[0071] Of the hardware configuration of the image processing device 1 according to this embodiment, the communication unit 13, the display unit 14, the operation unit 15, and the reading unit 16 are not essential. The communication unit 13 may not be used after it has been used once, for example, when acquiring the image processing program 1P and the machine learning library 1L stored in the storage unit 12 from an external server device. Similarly, the reading unit 16 may not be used after it has read and acquired the image processing program 1P and the machine learning library 1L from a storage medium. The communication unit 13 and the reading unit 16 may be the same device using serial communication such as USB.

[0072] The image processing device 1 may be configured as a Web server to provide the functions of the above-mentioned first machine learning model 111 and second machine learning model 112 to a Web client device including a display unit and a communication unit. In this case, the communication unit 13 is used to receive requests from the Web client device and transmit processing results.

[0073] The error used during learning may be an appropriate function such as square error, absolute value error, or cross entropy error depending on the input / output data and the learning purpose.

[0074] The present invention is not limited to the configuration of the above embodiment, and various modifications are possible within the scope of the invention. In order to achieve the object of the present invention, the present invention can be realized as an image processing method (learning method and inference method) having characteristic constituent means included in an image processing device (learning device and inference device) as steps, or as a program including those characteristic steps. The program can be stored in a ROM or the like, or can be distributed via a recording medium such as a USB memory or a communication network.

[0075] The present invention can also be realized as a computer system that transmits input data to an image processing device or a computer program and receives and uses output data from the image processing device or the computer program. This system is a processing system that uses data obtained from a machine learning model that has been trained by the above-mentioned processing, and can provide various services. The device used in this system is an image processing device equipped with a display unit and a communication unit, or an information processing device that can transmit and receive information to and from a computer, such as a so-called PC, a smartphone, a mobile terminal, or a game device. [Explanation of symbols]

[0076] 1 Image processing device (learning device and inference device) 12 Memory unit (learning result memory unit) 101 Learning processing execution unit 102 Inference processing execution unit 110 Image input unit 111 First Machine Learning Model 112 Second Machine Learning Model 113 Output section 200 Third Machine Learning Model 201 Text Encoder 301 Input image data 302 Output image data

Claims

1. A learning device that learns a setting value in a machine learning model based on image data, an image input unit that receives input image data and converts it into image data in which only the color components are downsampled while the brightness components are left as is; A first machine learning model that receives the converted input image data from the image input unit and generates compressed latent variables; A second machine learning model to which the latent variables generated by the first machine learning model are input and which outputs output image data; A learning process execution unit that executes a process of learning setting values ​​in the first machine learning model and the second machine learning model, The image input unit inputs the input image data to the first machine learning model, the learning process execution unit learns setting values ​​to convert the input image data into the latent variables using the first machine learning model, and to generate the output image data by restoring the input image data from the latent variables using the second machine learning model; A learning device characterized in that all image data of at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

2. 2. The learning device according to claim 1, wherein the latent variables follow an arbitrary probability distribution.

3. The learning device according to claim 1, characterized in that the image input unit inputs a brightness component of the input image data to an upper layer of the neural network of the first machine learning model, and inputs the downsampled color component to a lower layer than the upper layer.

4. The learning device according to claim 1, characterized in that the brightness component generated using the second machine learning model is output from an upper layer of the neural network of the second machine learning model, and the color component is output from a layer lower than the upper layer.

5. An inference device that executes a predetermined inference process on image data, an image input unit that accepts input of target image data; A first machine learning model to which the target image data is input from the image input unit; an inference processing execution unit that executes a predetermined inference processing on the target image data; The image input unit inputs the target image data to the first machine learning model, The inference device, characterized in that the inference processing execution unit converts the target image data into latent variables using a first machine learning model trained by a learning device described in any one of claims 1 to 4.

6. The inference device further comprises: A third machine learning model for image processing; a text encoder for incorporating user-entered text into the third machine learning model; The inference device according to claim 5, further comprising: a third machine learning model and a text encoder for processing latent variables output from the first machine learning model.

7. An inference device that executes a predetermined inference process on image data, a second machine learning model into which the latent variables are input; an inference processing execution unit that executes a predetermined inference processing using the second machine learning model, The inference processing execution unit generates output image data from latent variables using a second machine learning model trained by the learning device described in any one of claims 1 to 4. An inference device characterized in that:

8. The inference device further comprises: A third machine learning model for image processing; a text encoder for incorporating user-entered text into the third machine learning model; The inference device according to claim 7, characterized in that a latent variable processed using the third machine learning model and the text encoder is input to the second machine learning model to generate the output image data.

9. A computer program causing a computer to operate as the learning device according to any one of claims 1 to 4.

10. A computer program for causing a computer to operate as the inference device according to claim 5.

11. A computer program causing a computer to operate as the inference device according to claim 7.

12. A learning method executed by a learning device that learns a setting value in a machine learning model based on image data, comprising: An image input step of accepting input image data and converting it into image data in which only the color components are downsampled while the brightness components are left as is; A learning process execution step of executing a process of learning a setting value in a machine learning model, In the image input step, the converted input image data is input to a first machine learning model; In the learning process execution step, a setting value is learned so that the input image data is converted into a latent variable using the first machine learning model, and a setting value is learned so that output image data in which the input image data is restored from the latent variable is generated using the second machine learning model; A learning method characterized in that all image data of at least one of the input image data and the output image data is image data in which the brightness component is maintained and only the color component is downsampled.

13. An inference method executed by an inference device that executes a predetermined inference process on image data, comprising: an image input step of accepting input of target image data; An inference processing execution step of executing a predetermined inference processing on the target image data by using the first machine learning model, In the image input step, the target image data is input to the first machine learning model; An inference method, characterized in that in the inference processing execution step, the target image data is converted into latent variables using the first machine learning model trained by the learning method described in claim 11.

Images