Learning and reasoning devices

Abstract

To provide a learning apparatus configured to suppress computational complexity to efficiently generate image data with higher visual quality by using components based on visual characteristics of a person, in image data processing using machine leaning.SOLUTION: A learning apparatus includes: an input unit 110 for receiving an input of learning image data; a machine learning model 111 for receiving the learning image data as input; a learning processing execution unit 101 for executing processing for learning a setting value in the machine learning model 111; and a learning result storage unit for storing the setting value. The machine learning model includes at least a first section and a second section. In the first section, brightness components are learned. In the second section, color components are learned. In the first section, brightness components perceptible to human visual sensation are intensively learned and inferred, and hint data of the brightness components are used in the second section, thereby enabling execution of leaning for efficiently generating image data with higher visual quality in image processing using machine learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a learning device and an inference device that use a machine learning model for image processing. [Background technology]

[0002] Conventionally, a data processing method using machine learning has been known in which a data set serving as training data is provided to a computer program, and the program parameters are learned to generate a trained model capable of performing data processing on any data.

[0003] For example, in the process of "input image (teacher data) → learning program → output image (teacher data)" using machine learning, the parameters of the learning program are calculated so that the error between the input image and the output image is minimized, and a "trained model" is generated. Then, by using this "trained model", it is possible to generate an output image by inferring the output image from the input image, which is "input image (any data: for example, a low-resolution image) → trained model → output image (inference data: for example, a high-resolution image)".

[0004] In recent years, machine learning using neural networks has been applied to many fields. Especially in the fields of image and voice recognition, deep learning using neural networks in a multi-layer structure has demonstrated high recognition accuracy. In multi-layered deep learning, image processing is performed using a convolutional neural network (CNN), which uses multiple convolutional layers and pooling layers to extract input features.

[0005] Examples of image processing using neural networks include a super-resolution device that increases the resolution of signals (see, for example, Patent Document 1) and a diagnosis support device that facilitates grasping differences in disease areas and provides highly accurate diagnosis support (see, for example, Patent Document 2). There is also a machine learning model called AGGLLNet (see, for example, Non-Patent Document 1) that uses a data set with exposure correction and noise added, and can be used to brighten dark images, for example. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2020-27557 A [Patent Document 2] JP 2018-38789 A [Non-patent literature]

[0007] [Non-Patent Document 1] “AGLLNet: A machine learning model for brightening dark images, Kazuki Kyakuno” [Retrieved May 18, 2022], <URL:https: / / medium.com / axinc / agllnet-%E6%9A%97%E3%81%84%E7%94%BB%E5%83%8F%E3%82%92%E6%98%8E%E3%82%8B%E3%81%8F%E3%81%99%E3%82%8B%E6%A9%9F%E6%A2%B0%E5%AD%A6%E7%BF%92%E3%83%A2%E3%83%87%E3%83%AB-d59181ad89a9> Summary of the Invention [Problem to be solved by the invention]

[0008] In image data processing using a neural network, target image data is input to a machine learning model. For example, image data used in a machine learning model is represented as a collection of pixels (picture elements) partitioned by coordinates, and each pixel is assigned a pixel value (e.g., three values ​​of RGB color information). In the machine learning model, all pixel values ​​of the target image data are input to input neurons to execute machine learning.

[0009] Typically, when using image data expressed using RGB or YCbCr color space as learning / inference data for machine learning, machine learning is performed by setting the same number of channels on the input and output sides of the machine learning model as shown in Fig. 12. Specifically, three inputs and three outputs are set for RGB image data: R channel, G channel, and B channel, and for YCbCr image data: Y channel, Cb channel, and Cr channel.

[0010] However, human vision is much more sensitive to changes in brightness than to changes in color, and humans have visual characteristics that make it difficult for them to perceive something as unnatural even if the color components are reduced. Therefore, by ignoring such human visual characteristics and uniformly inputting / outputting all image data composed of multiple channels to a machine learning model as described above, it is believed that the learning processing efficiency decreases and the amount of calculation increases.

[0011] Furthermore, the image processing method described in the above patent document does not disclose any arithmetic processing of image data that takes into account the above-mentioned human visual characteristics. Furthermore, the machine learning model described in the above non-patent document does not reflect the difference in visual characteristics between brightness components and color components.

[0012] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a learning device and an inference device that can reduce the amount of calculations and efficiently generate visually better image data by using components based on human visual characteristics in image data processing using machine learning. [Means for solving the problem]

[0013] In order to achieve the above object, the present invention provides a learning device that receives input of predetermined learning image data, and learns setting values ​​of a machine learning model that is used to generate image data for output by performing inference processing with the machine learning model, the learning device including an input unit that receives input of learning image data, a machine learning model to which the learning image data is input from the input unit, a learning process execution unit that executes a process of learning setting values ​​in the machine learning model to be learned based on the learning image data using the machine learning model to be learned, and a learning result storage unit that stores the setting values, the machine learning model having at least a first section and a second section, and the learning process execution unit receives unprocessed image data of only a brightness component or unprocessed image data of brightness components and color components from the input unit, of When input to the machine learning model, an inference process is performed in the first section, and the inferred image data of the brightness component output from the first section is compared with the processed image data which is the teacher data for the brightness component. The learning process execution unit inputs the unprocessed image data of only the color components from the input unit and the inferred image data of the brightness component which is output from the first section to the second section of the machine learning model, and performs an inference process using the input data, and compares the inferred image data of the color components output from the second section with the processed image data which is the teacher data for the color components, or compares the inferred image data of the brightness component and color components output from the second section with the processed image data which is the teacher data for the brightness component and color components, and learns parameter setting values ​​of the machine learning model through each of the comparisons.

[0014] In this learning device, it is preferable that the learning process execution unit outputs inference image data including components other than brightness components from the first section to the second section, and learns parameter setting values ​​of a machine learning model.

[0015] In this learning device, it is preferable that the learning process execution unit outputs inference image data including components other than brightness components from the first section to the second section, and the second section performs inference processing using only the data output from the first section, thereby learning parameter setting values ​​of the machine learning model.

[0016] In order to achieve the above object, the present invention provides an inference device for executing a predetermined inference process on target image data using a machine learning model and generating image data for output, the inference device comprising: an input unit that accepts input of target image data; a machine learning model to which the target image data is input from the input unit; and an inference process execution unit that executes a predetermined inference process on the target image data using the machine learning model that executes the inference process, the machine learning model having at least a first partition and a second partition, and the inference process execution unit receives target image data consisting of only brightness components or target image data consisting of brightness components and color components from the input unit, of When input, an inference process is performed in the first section, and inference image data of the brightness component is output from the first section to the second section, and the inference process execution unit performs inference process in the second section using target image data of only the color component from the input unit and the inference image data of the brightness component output from the first section, and outputs inference image data of the color component or inference image data of the brightness component and the color component from the second section.

[0017] In this inference device, it is preferable that the inference processing execution unit outputs inference image data including components other than brightness components from the first section to the second section, and the second section performs inference processing using the data output from the first section.

[0018] In this inference device, it is preferable that the inference process execution unit outputs inference image data including a brightness component and a color component from the second section.

[0019] It is preferable that this inference device is characterized by comprising an image combination unit that combines inferred image data of the brightness component output from the first section and inferred image data of the color component output from the second section.

[0020] The present invention also provides a computer program for causing a computer to operate as the above-described learning device or inference device.

[0021] The present invention also provides a learning method for learning setting values ​​in a machine learning model used to generate image data for output based on predetermined learning image data, the method comprising: an input step of accepting input of learning image data; Accepted at Using a machine learning model to which training image data is input, , studies The method includes a learning process execution step for executing a process to learn setting values ​​in a machine learning model to be trained, and a learning result storage step for storing the setting values, wherein the machine learning model has at least a first section and a second section, and in the learning process execution step, learning processing is performed in the first section for only the brightness component of the input learning image data, and in the second section, learning processing is performed by inputting inference image data of the brightness component processed in the first section and the color components of the input learning image data.

[0022] The present invention also provides an inference method for executing a predetermined inference process on target image data using a machine learning model to generate image data for output, the method comprising: an input step of accepting an input of the target image data; Accepted atThe method includes an inference processing execution step in which target image data is input and a predetermined inference processing is performed on the target image data using a machine learning model that executes inference processing, the machine learning model having at least a first section and a second section, and in the inference processing execution step, inference processing is performed in the first section only on the brightness component of the input target image data, and in the second section, inference image data of the brightness component processed in the first section and the color components of the input target image data are input and inference processing is performed on image data for output. Effect of the Invention

[0023] The learning device according to the present invention includes an input unit that accepts input of learning image data, a machine learning model to which the learning image data is input, a learning process execution unit that executes a process of learning setting values ​​in the machine learning model, and a learning result storage unit that stores the setting values, the machine learning model having at least a first section and a second section, and learning and inference of a brightness component is performed in the first section, and learning and inference of a color component is performed in the second section. With this configuration, the learning device according to the present invention utilizes the characteristic that image data basically contains a lot of information in the brightness component, and focuses on learning and inferring the brightness component to which human vision is sensitive, and can generate visually good image data even if the amount of calculation used for learning and inferring the color components that do not affect human vision is relatively reduced. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of the image processing device. [Diagram 3] FIG. 2 is an image diagram of the learning / inference process in the machine learning model provided in the image processing device. [Figure 4] 4 is a flowchart showing an operation procedure during a learning operation of the image processing device. [Diagram 5] 4 is a flowchart showing an operation procedure during an inference operation of the image processing device. [Figure 6] FIG. 11 is an image diagram of a learning / inference process in a machine learning model provided in an image processing device according to a second embodiment of the present invention. [Figure 7] 11A to 11C are conceptual diagrams showing variations of inference processing in a machine learning model provided in an image processing device according to embodiment 3 of the present invention. [Figure 8] FIG. 13 is an image diagram showing another variation of the inference process in the machine learning model provided in the image processing device according to the fourth embodiment of the present invention. [Figure 9] FIG. 13 is an image diagram of a learning / inference process in a machine learning model provided in an image processing device according to a fifth embodiment of the present invention. [Figure 10] 13 is a conceptual diagram showing variations of inference processing in a machine learning model provided in an image processing device according to a sixth embodiment of the present invention. FIG. [Figure 11] FIG. 13 is an image diagram of a learning / inference process in a machine learning model provided in an image processing device according to a seventh embodiment of the present invention. [Figure 12] This is an image of learning / inference data for conventional machine learning. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] (Embodiment 1) An image processing device according to a first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 5. In the first embodiment, the image processing device functions as at least one of a learning device that learns setting values ​​in a machine learning model based on predetermined learning image data, and an inference device that executes a predetermined inference process on target image data using a machine learning model.

[0026] First, each processing unit of the image processing device 1 will be described with reference to Fig. 1. As shown in Fig. 1, the image processing device 1 includes a control unit 10, an image processing unit 11, a storage unit 12, a communication unit 13, a display unit 14, an operation unit 15, and a reading unit 16. Note that, although the image processing device 1 and the operation in the image processing device 1 will be described below as a single server computer, the image processing device 1 may be configured to distribute processing among multiple computers.

[0027] The control unit 10 uses a processor such as a CPU and a memory to control the components of the device to realize various functions. The image processing unit 11 uses a processor such as a GPU or a dedicated circuit and a memory to execute image processing in response to control instructions from the control unit 10. The control unit 10 and the image processing unit 11 may be configured as a single piece of hardware (SoC: System on a Chip) that integrates processors such as a CPU and a GPU, memory, and further a storage unit 12 and a communication unit 13.

[0028] The storage unit 12 uses a hard disk or a flash memory. The storage unit 12 stores an image processing program 1P and a machine learning library 1L that functions as a machine learning model (e.g., CNN). The storage unit 12 also stores definition data that defines the machine learning model, parameters including setting values ​​in the trained machine learning model, and the like.

[0029] The communication unit 13 is a communication module that realizes a communication connection to a communication network such as the Internet, etc. The communication unit 13 uses a network card, a wireless communication device, or a carrier communication module.

[0030] The display unit 14 uses a liquid crystal panel, an organic EL (Electro Luminescence) display, etc. The display unit 14 is capable of displaying an image by processing in the image processing unit 11 according to an instruction from the control unit 10.

[0031] The operation unit 15 includes a user interface such as a keyboard or a mouse. It may be a physical button provided on the housing, or a software button displayed on the display unit 14. The operation unit 15 notifies the control unit 10 of operation information by the user.

[0032] The reading unit 16 can use, for example, a disk drive to read the image processing program 2P and the machine learning library 3L stored in the recording medium 2, which is an optical disk or the like. The image processing program 1P and the machine learning library 1L stored in the storage unit 12 may be the image processing program 2P and the machine learning library 3L read by the reading unit 16 from the recording medium 2 and copied into the storage unit 12 by the control unit 10.

[0033] Next, the image processing function of the image processing device 1 will be described with reference to Fig. 2. The control unit 10 of the image processing device 1 includes a learning process execution unit 101 and an inference process execution unit 102. The learning process execution unit 101 functions as a machine learning model (machine learning engine) based on the machine learning library 1L, definition data, and parameter information stored in the storage unit 12. That is, the learning process execution unit 101 executes a process of learning setting values ​​(parameters, etc.) in the machine learning model to be learned based on the learning image data, using the machine learning model to be learned.

[0034] The inference processing execution unit 102 performs image processing based on the image processing program 1P stored in the storage unit 12. That is, the inference processing execution unit 102 executes a predetermined inference process on the input target image data using a machine learning model. The inference processing execution unit 102 also functions as an image processing execution unit that inputs image data, which is input data, to the input unit 110 based on a user's operation using the operation unit 15.

[0035] In the first embodiment, the input unit 110 of the image processing unit 11 accepts input of learning image data or target image data to be inferred. The learning process execution unit 101 outputs the learning image data from the input unit 110 to a machine learning model 111. The machine learning model 111 includes a first partition 111a that processes a brightness component (Y component) and a second partition 111b that processes color components (CbCr components). In this configuration, the inferred brightness component and the inferred color component can be output separately from different partitions (different nodes) within a single machine learning model.

[0036] The learning process execution unit 101 outputs the learning image data input from the input unit 110 to the first section 111a and the second section 111b of the machine learning model 111. When performing inference processing, the inference process execution unit 102 outputs the inference target image data to the first section 111a and the second section 111b of the machine learning model 111. The type of each image data can be determined, for example, by reading information or an identifier in the header section of the image data. When the machine learning model at the time of inference is CNN, the machine learning model 111 may include multiple stages of convolution layers and pooling layers defined by definition data, and a fully connected layer, extracts features of the image data, and performs image processing based on the extracted features.

[0037] The image data processed by the machine learning model 111 is output to the output unit 112. The output unit 112 includes an image combination unit 112a, and can input the data output from the machine learning model 111 to the image combination unit 112a to execute image combination processing. In addition, when the output data processed by the machine learning model 111 is YCbCr or YUV, a process of converting it to RGB can also be performed. Note that the output data may be rendered as an image and output to the display unit 14.

[0038] During inference processing, the machine learning model 111 performs optimization processing of image data (e.g., processing of increasing image resolution and removing noise) based on parameters that have already been learned. The image combining unit 112a can combine the image data generated by the machine learning model 111 by overlay processing or the like, and output the output data to the storage unit 12.

[0039] Next, a specific example of the learning / inference process in the first machine learning model 111 will be described with reference to Fig. 3. In the following description, pre-processed image data refers to data input to the machine learning model 111 during learning, processed image data refers to data (teacher data) that is compared with the output of the machine learning model 111 during learning, target image data refers to data input to the machine learning model 111 during inference, and inference image data refers to data output from the machine learning model 111 during learning or inference.

[0040] In the first embodiment, the learning image data input to the machine learning model 111 is composed of unprocessed image data 301 (301a, 301b, 301c) and processed image data 302 (302a, 302b, 302c) which is teacher data. The unprocessed image data 301 is composed of at least one of image data 301a having brightness components and color components (image data having a Y component and color difference components of Cb and Cr in this figure), image data 301b having only color components (image data having only color difference components of Cb and Cr in this figure), and image data 301c having only brightness components (image data having only a Y component in this figure). The processed image data 302 is teacher data to be compared with inferred image data inferred by inputting the unprocessed image data to the machine learning model 111, and various parameters of the machine learning model 111 are adjusted so that the inferred image data approaches the teacher data. For example, when the machine learning model 111 is used to generate high-resolution image data from low-resolution image data, the pre-processed image data is low-resolution image data, the inferred image data is high-resolution image data inferred by the machine learning model 111, and the processed image data is high-resolution image data that is teacher data. Therefore, like the data inferred by the machine learning model 111, the processed image data is composed of image data 302a having only a brightness component, image data 302b having only a color component, and image data 302c having both a brightness component and a color component.

[0041] In the description of the first embodiment, the brightness component refers to a component related to the luminance, brightness, density, etc. of image data, and is typically the brightness component Y in YCbCr. In addition to Y in YCbCr, there are L in Lab color space, V in HSV color space, L in HLS color space, etc. YUV or YPbPr may be used instead of YCbCr. Furthermore, the color component typically refers to the color difference component in YCbCr, or the hue and saturation in HSV. In addition, although an RGB image or the like is a mixture of brightness and color components, when it is used as image data for learning and inference, it is possible to convert it to YCbCr or the like and then use all or a part of the components. In this case, it is assumed that the image processing device 1 is provided with a function or a processing unit for color conversion from an RGB image to YCbCr, YUV, etc.

[0042] During learning of the machine learning model 111, the learning process execution unit 101 executes a process of learning the setting values ​​in the machine learning model 111 based on the unprocessed image data 301 and the processed image data 302, as shown in Fig. 3. In this machine learning model 111, unprocessed image data 301a (three channel input of Y channel, Cb channel, and Cr channel) having brightness components and color components, unprocessed image data 301b (two channel input of Cb channel and Cr channel) having only color components, and unprocessed image data 301c (one channel input of Y channel) having only brightness components are set. Note that, among the unprocessed image data 301a, the color components (CbCr) may be input to the first section 111a of the machine learning model 111, and only the color components (CbCr) may be directly input to the second section using a skip connect.

[0043] On the other hand, the output of the machine learning model 111 is set to inferred image data 304c having brightness and color components (three-channel output: Y channel, Cb channel, and Cr channel), inferred image data 304b having only color components (two-channel output: Cb channel and Cr channel), and inferred image data 304a having only the brightness channel (one-channel output: Y channel).

[0044] In this way, in the machine learning model 111, at least one or more input channels are set to receive image data having brightness and color components, only color components, or only brightness components, and at least one or more output channels are set to receive image data having brightness and color components, only color components, or only brightness channels. During learning of the machine learning model 111, the learning process execution unit 101 inputs the unprocessed image data 301c or the unprocessed image data 301a to the machine learning model 111, and executes a process of learning the setting values ​​in the machine learning model 111 based on a comparison result between the inference image data 304a output from the first section 111a and the processed image data 302a as the teacher data, and a comparison result between the inference image data 304b or 304c output from the second section 111b and the processed image data 302b or 302c as the teacher data.

[0045] In this way, by providing two sections 111a and 111b within one machine learning model 111, with the first section 111a learning exclusively the brightness component (Y component) and the second section 111b learning exclusively the color components (CbCr), it is possible to perform learning that reflects the visual characteristics of people who are more sensitive to changes in brightness than to changes in color components.

[0046] On the other hand, during inference by the machine learning model 111, target image data 303a (YCbCr), 303b (CbCr), or 303c (Y) can be input to the machine learning model 111, and inference image data 304a (Y) can be obtained from the first section 111a, and inference image data 304b (CbCr) or 304c (YCbCr) can be obtained as the output of the second section 111b. This target image data 303 may be image data expressed using YCbCr, image data expressed using only color components (CbCr), or image data expressed using only brightness component (Y), or image data expressed using RGB converted into YCbCr.

[0047] Next, an operation procedure when the image processing device 1 according to the first embodiment operates as a learning device will be described with reference to FIG. 4. First, when image data is input to the input unit 110 (Yes in S401), the input unit 110 outputs the learning image data to the machine learning model 111 (S402). Next, the input learning image data (unprocessed image data) is processed by the machine learning model 111 to generate inference image data 304a, 304b, and 304c, and these inference image data are compared with the processed image data 302a, 302b, and 302c, which are teacher data, and various parameters of the machine learning model 111 are updated (updating process of setting values) (S403). This updating of setting values ​​is a process of minimizing the difference between the input data and the answer data, for example, by updating the parameters by a mini-batch gradient descent method. Next, the updated setting values ​​are stored in the storage unit 12 (S404).

[0048] The parameters of the first section 111a of the machine learning model 111 may be updated only by comparing the inference image data 304a and the processed image data 302a, or may be updated based on the comparison of the inference image data 304a and the processed image data 302a, as well as the comparison of the inference image data 304b and the processed image data 302b or the comparison of the inference image data 304c and the processed image data 302c. The parameters of the second section 111b may also be updated based on the comparison of the inference image data 304b and the processed image data 302b or the comparison of the inference image data 304c and the processed image data 302c, or may be updated based on the comparison of the inference image data 304a and the processed image data 302a in addition to these comparisons.

[0049] Next, an operation procedure when the image processing device 1 according to the first embodiment operates as an inference device will be described with reference to Fig. 5. First, when target image data is input to the input unit 110 (Yes in S501), the input unit 110 inputs the target image data to the machine learning model 111 (S502). Next, the machine learning model 111 performs inference processing using CNN or the like on the input target image data to generate inference image data (S503). Then, the inference result is stored in the storage unit 12 (S504).

[0050] As described above, in the image processing device 1 according to the first embodiment, a first section 111a that processes only the brightness component is set in the front stage of the processing of the machine learning model 111, a brightness component (Y component) is output from a node of any layer of the processing layer of the first section 111a, and a second section 111b that processes only the color component is set in the rear stage of the processing of the machine learning model 111. Specifically, the learning process execution unit 101 executes a process that minimizes the error between inference image data obtained by inputting unprocessed image data to the machine learning model 111 and known processed image data, and can update the parameters (weights). The parameters obtained by this learning process are stored in the storage unit 12. At this time, the storage unit 12 functions as a learning result storage unit. On the other hand, when functioning as an inference device, the definition data and parameters stored in the storage unit 12 are used to generate inference image data using the machine learning model 111.

[0051] For this reason, in machine learning, the image processing device 1 utilizes the characteristic that image data basically contains a lot of information in the brightness component, and can generate visually good image data by focusing on learning and inferring the brightness component to which human vision is sensitive, and relatively reducing the amount of calculation used for learning and inferring color components that do not affect human vision. Here, "focusing on learning and inferring the brightness component" means that the processing amount of the first section 111a for learning and inferring the brightness component is set to be greater than the processing amount of the second section 111b, or by inputting the brightness component to the first section 111a, the processing of the brightness component is focused on first.

[0052] That is, the brightness component and the color component can be input / output to the machine learning model 111 separately, and the processing amount can be changed for each channel to provide different accuracy, or the data from the channel processed in the first section 111a can be used as hint data for the subsequent processing in the second section 111b, thereby obtaining optimal inferred image data while suppressing the amount of calculation. In this way, by first processing the brightness component in the first section 111a to create hint data, compared to the case where image data (YCbCr) consisting of the brightness component and color component are processed together in the machine learning model, when the image data (Y) of the brightness component is processed first to create hint data, the latter has better accuracy when compared with the same amount of calculation, and as a result, it is possible to improve the quality of the output image.

[0053] (Embodiment 2) Hereinafter, a second embodiment of the image processing device 1 according to the present invention will be described with reference to Fig. 6. In the configuration of the image processing device 1 according to the present embodiment, the same components as those of the image processing device 1 according to the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted. 6, image data 304d other than the inference image data 304a generated in the first section 111a is used as data to be handed over from the first section 111a to the second section 111b of the machine learning model 111. This image data 304d can be extracted from nodes before and after (not limited to the nodes immediately before or after) the node in the first section 111a where the inference image data 304a is generated.

[0054] Learning of the first section 111a is performed in the same manner as in the first embodiment, and inference image data 304a having only the brightness component extracted from the first section 111a is compared with the processed image data 302a, which is the teacher data. On the other hand, learning of the second section 111b is performed using unprocessed image data 301b having only the color components and image data 304d delivered from the first section 111a, and inference image data 304b and 304c, which are the output of the second section 111b, are compared with the processed image data 302b and 302c, which are the teacher images, to learn the parameter setting values ​​for each section of the machine learning model 111.

[0055] During inference, target image data 303a (Y, CbCr) or 303c (Y) is input to the first section 111a of the machine learning model 111, and inference image data 304a is generated from the first section 111a. Also, image data 304d (image data other than the inference image data 304a) delivered from the first section 111a and target image data 303b (CbCr) are input to the second section 111b, and inference image data 304c having brightness components and color components, or inference image data 304b having only color components is generated. Then, when the output from the second section 111b is inference image data 304b (CbCr) having only color components, the inference image data 304a (Y) extracted from the first section 111a is combined with the data by the image combining section 112a of the output section 112, and YCbCr data is generated.

[0056] That is, in the above-mentioned embodiment 1, the data handed over from the first section 111a to the second section 111b is the inference image data 304a (Y) having only the brightness component, whereas in the present embodiment 2, data (other than Y) obtained by branching from a layer preceding the layer (node) in which the inference image data 304a is generated in the first section 111a, and image data 304d (other than Y) obtained by further processing the output of the layer (node) in which the inference image data 304a is generated are used. This image data 304d is one-channel data consisting of a Y' component other than the Y component, or multi-channel data including a Y component. That is, in the present embodiment 2, the output of the inference image data 304a having only the brightness component is set in the middle of the machine learning model 111, and inference is performed in the second section 111b using the image data 304d other than the inference image data 304a.

[0057] With this configuration, in addition to the effects of the first embodiment, other than the inference image data 304a itself consisting only of brightness components (for example, data from a processing stage prior to being output as the inference image data 304a, or data from further processing of the inference image data 304a) can be input to the second section 111b as hint data processed in the first section 111a, and image data that is visually good can be generated by focusing on learning and inferring brightness components to which human vision is sensitive and relatively reducing the amount of calculations used for learning and inferring color components that do not affect human vision. The unprocessed image data 301b or target image data 303b input to the second section 111b may be configured to supply CbCr data having color components from the unprocessed image data 301a or target image 303a input to the first section 111a by skip connect.

[0058] (Embodiment 3) Hereinafter, a third embodiment of the image processing device 1 according to the present invention will be described with reference to FIG. 7. The third embodiment is one form of inference in the first embodiment shown in FIG. 3. That is, during learning, learning is performed using the inference image data 304a from the first section 111a and the processed image data 302a in the same manner as in the first embodiment shown in FIG. 3, and during inference, the target image data 303a consisting of the brightness component and the color component is input to the machine learning model 111, and the inference image data 304a (Y) consisting of only the brightness component inferred in the first section 111a is output to the second section 111b, and in the second section 111b, the inference image data 304a (Y) and the target image data 303b (CbCr) are used to infer inference image data 304c (YCbCr). That is, in the third embodiment, one branch of the output of the inference image data 304a having only the brightness component generated during inference of the first section 111a is omitted. As a result, the inference image data 304c (YCbCr image) having the brightness component and the color component can be output as the inference result of the machine learning model 111.

[0059] Moreover, since the inferred image data 304a having only the brightness component is not branched and output, the output from the second section must be inferred image data 304c having the brightness component and the color component. In this third embodiment as well, the brightness component to which human vision is sensitive is learned and inferred with a focus, and the amount of calculations used for learning and inferring the color component that does not affect human vision is relatively reduced, thereby generating visually good image data. Note that the target image data 303b input to the second section 111b may be configured to supply CbCr data having color components from the target image 303a input to the first section 111a by skip connect.

[0060] (Embodiment 4) Hereinafter, a fourth embodiment of the image processing device 1 according to the present invention will be described with reference to FIG. The third embodiment is another form of inference in the first embodiment shown in FIG. 3. That is, during learning, learning is performed using the inference image data 304a from the first section 111a and the processed image data 302a, as in the first embodiment shown in FIG. 3, and during inference, inference image data 304a (Y) consisting of only the brightness component inferred in the first section 111a is output to the second section 111b, and in the second section 111b, inference image data 304c (YCbCr) is inferred using the inference image data 304a (Y) and the target image data 303b (CbCr). That is, in the third embodiment, the target image data 303c consisting of only the brightness component is input to the first section 111a of the machine learning model 111, and inference image data 304c is obtained. At this time, one branch of the output of the inference image data 304a having only the brightness component generated during the inference of the first section 111a is omitted. As a result, inference image data 304c (YCbCr image) having the brightness component and the color component can be output as the inference result of the machine learning model 111. Moreover, since the inferred image data 304a having only the brightness component is not branched and output, the output from the second section must be inferred image data 304c having the brightness component and the color component. In this third embodiment as well, it is possible to generate visually pleasing image data by focusing on learning and inferring the brightness component to which human vision is sensitive, and relatively reducing the amount of calculations used in learning and inferring the color components that have little effect on human vision.

[0061] (Embodiment 5) A fifth embodiment of the image processing device 1 according to the present invention will be described below with reference to Fig. 9. The fifth embodiment is an embodiment in which the input to the first section 111a of the machine learning model 111 in the second embodiment shown in Fig. 6 is image data (YCbCr) having a brightness component and a color component.

[0062] In the fifth embodiment, the learning process execution unit 101 inputs pre-processed image data 301a having brightness and color components to the first section 111a during learning, and generates inferred image data 304a of the brightness component in the first section 111a. This inferred image data 304a is compared with the processed image data 302a, and the parameter setting value of each section of the machine learning model 111 is set. In addition, the data transferred from the first section 111a to the second section 111b is image data 304d other than the inferred image data 304a. This image data 304d is multi-channel data including a Y component. This image data 304d is input to the second section 111b, and inferred image data 304b having only color components or inferred image data 304c having brightness and color components is obtained, and these inferred image data 304b or 304c are compared with the processed image data 302b or 302c, and the parameter setting value of each section of the machine learning model 111 is set.

[0063] In this embodiment, during inference, target image data 303a (Y, CbCr) is input to the first section 111a of the machine learning model 111, inference image data 304a is generated from the first section 111a, and image data 304d (image data other than the inference image data 304a) delivered from the first section 111a is input to the second section 111b, and inference image data 304c having brightness components and color components, or inference image data 304b having only color components is generated. Then, when the output from the second section 111b is inference image data 304b (CbCr) having only color components, the inference image data 304a (Y) extracted from the first section 111a is combined with the data by the image combining section 112a of the output section 112 to generate YCbCr data. In this embodiment, the brightness component Y is learned for both the first section 111a and the second section 111b, and the brightness component to which human vision is sensitive is primarily learned and inferred, while the amount of calculations used in learning and inferring color components that have little effect on human vision is relatively reduced, thereby generating visually pleasing image data.

[0064] (Embodiment 6) Hereinafter, a sixth embodiment of the image processing device 1 according to the present invention will be described with reference to FIG. 10. The fifth embodiment is a diagram showing the process during inference in the fourth embodiment shown in FIG. 9. During inference, when target image data 303a having brightness and color components is input to the machine learning model 111, the inference process execution unit 102 transfers image data 304d, which is multi-channel data including brightness components, from the first section 111a that has been learned with a focus on the brightness component to the second section 111b, and further infers the brightness and color components in the second section 111b to obtain inferred image data 304c. By inferring the brightness component in both the first section 111a and 111b without using the inferred image data 304a of the brightness component inferred in the first section 111a as in this embodiment, it is possible to infer the brightness component to which human vision is sensitive with a focus, and generate visually pleasing image data.

[0065] (Embodiment 7) A seventh embodiment of the image processing device 1 according to the present invention will be described below with reference to Fig. 11. In this embodiment, unlike the first to fifth embodiments, the machine learning model 111 prioritizes the second section 111b that processes color components, rather than the first section 111a that processes brightness components.

[0066] Specifically, during learning of the machine learning model 111, the learning process execution unit 101 processes unprocessed image data 301a having brightness and color components using the second section 111b of the machine learning model 111, compares inference image data 304e (CbCr) consisting of color components output from the second section 111b with processed image data 302b which is teacher data for the color components, and learns the setting values ​​of parameters of the second section 111b and the first section 111a of the machine learning model 111. Also, the inference image data 304e is delivered from the second section 111b to the first section 111a, and the inference image data 304e and unprocessed image data 301c which is image data of only the brightness component are processed in the first section 111a. Inferred image data 304c consisting of brightness and color components is output from the first section 111a, and this inferred image data 304c is compared with processed image data 302c, which is teacher data for the brightness and color components, to learn the parameter setting values ​​of the second section 111b and the first section 111a of the machine learning model 111. In this embodiment, the amount of calculations contributing to the learning and inference of the brightness component in the first section 111a is increased to focus on learning and inferring the brightness component, which is sensitive to human vision. Note that the input of the unprocessed image data 301c having only the brightness component to the first section 111a may be such that only the brightness component is supplied from the unprocessed image data 301a by skip connect.

[0067] During inference by the machine learning model 111, target image data 303a is input to the second section 111b, and inference image data 304e consisting of color components output from the second section 111b and target image data 303c consisting of only the brightness component are input to the first section 111a, and processing is performed to generate inference image data 304c consisting of brightness and color components.

[0068] In this manner, in the sixth embodiment, the color components are first processed in the second section 111b, and the inference image data 304e having the color components serving as hint data can be input to the first section 111a. In this case as well, the learning of the brightness component, the learning of the brightness component of the first section 111a where inference is performed, and the amount of calculations contributing to inference are increased, so that the learning and inference are focused on the brightness component to which human vision is sensitive.

[0069] Of the hardware configuration of the image processing device 1 according to this embodiment, the communication unit 13, the display unit 14, the operation unit 15, and the reading unit 16 are not essential. The communication unit 13 may not be used after it has been used once, for example, when acquiring the image processing program 1P and the machine learning library 1L stored in the storage unit 12 from an external server device. Similarly, the reading unit 16 may not be used after it has read and acquired the image processing program 1P and the machine learning library 1L from a storage medium. The communication unit 13 and the reading unit 16 may be the same device using serial communication such as USB.

[0070] The image processing device 1 may be configured as a Web server to provide the function of the above-mentioned machine learning model 111 to a Web client device including a display unit and a communication unit. In this case, the communication unit 13 is used to receive requests from the Web client device and transmit processing results.

[0071] The error used during learning may be an appropriate function such as squared error, absolute error, or cross entropy error depending on the input and output data and the learning purpose. For example, if the output is classification, cross entropy error is used. Flexible operation can be applied, such as using other criteria instead of using an error function. This error function itself may be evaluated using an external machine learning model.

[0072] The present invention is not limited to the configuration of the above embodiment, and various modifications are possible within the scope of the invention. In order to achieve the object of the present invention, the present invention can be realized as an image processing method (learning method and inference method) having characteristic constituent means included in an image processing device (learning device and inference device) as steps, or as a program including those characteristic steps. The program can be stored in a ROM or the like, or can be distributed via a recording medium such as a USB memory or a communication network.

[0073] The present invention can also be realized as a computer system that transmits input data to an image processing device or a computer program and receives and uses output data from the image processing device or the computer program. This system is a processing system that uses data obtained from a machine learning model that has been trained by the above-mentioned processing, and can provide various services. The device used in this system is an image processing device equipped with a display unit and a communication unit, or an information processing device that can transmit and receive information to and from a computer, such as a so-called PC, a smartphone, a mobile terminal, or a game device. [Explanation of symbols]

[0074] 1 Image processing device (learning device and inference device) 12 Memory unit (learning result memory unit) 101 Learning processing execution unit 102 Inference processing execution unit 110 Input section 111 Machine Learning Models 111a First Section 111b Second Section 112 Output section 112a Image combiner 301 Unprocessed image data 301a Unprocessed image data having brightness and color components 301b Unprocessed image data having only color components 301c Unprocessed image data having only brightness components 302 Processed image data (teaching data) 302a Processed image data having only brightness components 302b Processed image data having only color components 302c Processed image data having brightness and color components 303 Target image data 303a Object image data having brightness and color components 303b Object image data having only color components 303c Object image data having only brightness components 304 Inference Image Data 304a Inference image data having only brightness components 304b Inferred image data having only color components 304c Inferred image data having brightness and color components 304d Inferred image data other than inferred image data having only the brightness component inferred in the first section 304e Inferred image data having only the color components inferred in the second section

Claims

1. A learning device that learns setting values ​​of a machine learning model used to input predetermined learning image data, perform inference processing with the machine learning model, and generate output image data, An input unit that accepts input of learning image data; A machine learning model to which learning image data is input from the input unit; A learning process execution unit for executing a process of learning a setting value in the machine learning model to be learned based on the learning image data by using the machine learning model to be learned; a learning result storage unit that stores the setting value, The machine learning model has at least a first partition and a second partition; The learning process execution unit performs an inference process in the first section when unprocessed image data consisting of only a brightness component or unprocessed image data consisting of a brightness component and a color component is input from the input unit to the machine learning model, and compares the inference image data of the brightness component output from the first section with the processed image data which is the teacher data of the brightness component, and The learning process execution unit inputs the unprocessed image data of only the color components from the input unit and the inferred image data of the brightness component output from the first section to the second section of the machine learning model, performs inference processing using the input data, and compares the inferred image data of the color components output from the second section with the processed image data that is the teacher data for the color components, or compares the inferred image data of the brightness component and the color components output from the second section with the processed image data that is the teacher data for the brightness component and the color components, A learning device that learns parameter setting values ​​of a machine learning model through each of the comparisons.

2. The learning device according to claim 1, characterized in that the learning process execution unit outputs inference image data including components other than brightness components from the first section to the second section, and learns parameter setting values ​​of a machine learning model.

3. The learning device according to claim 1, characterized in that the learning process execution unit outputs inference image data including components other than brightness components from the first section to the second section, and the second section performs inference processing using only the data output from the first section, thereby learning parameter setting values ​​of a machine learning model.

4. An inference device for executing a predetermined inference process on target image data using a machine learning model and generating image data for output, An input unit that accepts input of target image data; A machine learning model to which target image data is input from the input unit; An inference processing execution unit that executes a predetermined inference processing on the target image data using a machine learning model that executes the inference processing, The machine learning model has at least a first partition and a second partition; The inference processing execution unit performs an inference process in the first section when target image data consisting of only a brightness component or target image data consisting of a brightness component and a color component is input from the input unit, and outputs inference image data of the brightness component from the first section to the second section, The inference processing execution unit performs inference processing in the second section using target image data of only the color components from the input unit and inference image data of the brightness component output from the first section, and outputs inference image data of the color components or inference image data of the brightness component and the color components from the second section.

5. The inference device according to claim 4, characterized in that the inference processing execution unit outputs inference image data including components other than brightness components from the first section to the second section, and the second section performs inference processing using the data output from the first section.

6. 6. The inference device according to claim 5, wherein the inference process execution unit outputs inference image data including a brightness component and a color component from the second section.

7. 5. The inference device according to claim 4, further comprising an image combining unit that combines inferred image data of the brightness component output from the first section and inferred image data of the color component output from the second section.

8. A computer program causing a computer to operate as the learning device or the inference device according to any one of claims 1 to 7.

9. A learning method for learning a setting value in a machine learning model used to generate image data for output based on predetermined learning image data, comprising: An input step of accepting input of learning image data; a learning process execution step for executing a process of learning a setting value in the machine learning model to be learned based on the learning image data by using the machine learning model to which the learning image data received in the input step is input; a learning result storage step of storing the setting value, The machine learning model comprises at least a first partition and a second partition; A learning method characterized in that, in the learning process execution step, learning process is performed in the first section for only the brightness component of the input learning image data, and in the second section, learning process is performed by inputting the inferred image data of the brightness component processed in the first section and the color components of the input learning image data.

10. An inference method for performing a predetermined inference process on target image data using a machine learning model to generate image data for output, comprising: An input step of accepting input of target image data; an inference processing execution step in which the target image data received in the input step is input, and a predetermined inference processing is executed on the target image data using a machine learning model that executes the inference processing; The machine learning model comprises at least a first partition and a second partition; An inference method characterized in that, in the inference processing execution step, inference processing is performed in the first section only on the brightness component of the input target image data, and in the second section, the inference image data of the brightness component processed in the first section and the color components of the input target image data are input and image data for output is inferred.

Images