Image processing method, image processing device, and program

Abstract

To obtain an output image with high resolution in image processing using a machine learning model.SOLUTION: An image processing method includes: a step S203 of dividing a first gray scale image 21 to create a plurality of second gray scale images with a lower resolution than that of the first gray scale image 21; and an estimation step S204 of inputting the plurality of second gray scale images 23 to a machine learning model to create a plurality of upscaled third gray scale images 24.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to image processing using machine learning models. [Background technology]

[0002] Patent Document 1 discloses an image processing method for identifying the features of a color image by converting a color image represented by RGB to YUV and inputting information on the high-frequency components of the resulting Y image (luminance image) into a machine learning model. The image processing method in Patent Document 1 uses a Convolutional Neural Network (CNN) as the machine learning model, which generates an output image by convolving a filter with an input image multiple times. Furthermore, the image processing method in Patent Document 1 uses a reduced luminance image as the input image, thereby reducing the amount of calculation in the CNN and speeding up processing. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-175107 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the input image in Patent Document 1 is a reduced luminance image, which has a lower resolution than the luminance image before being reduced. Therefore, it is difficult to obtain a high-resolution output image with the image processing method in Patent Document 1.

[0005] Therefore, an object of the present invention is to obtain a high-resolution output image in image processing using a machine learning model. [Means for solving the problem]

[0006] The image processing method of the present invention is to, based on a first color image acquired by photography, select a color image having a larger number of pixels than the first color image. big An image processing method for generating a second color image, the method comprising: generating a first grayscale image from a first color image; and dividing the first grayscale image to generate a second grayscale image having a pixel count of 1 / 2. small generating a plurality of second grayscale images; and ISO sensitivity information for shooting of Connected to the channel By inputting it into a machine learning model, upscaled to the plurality of second grayscale images. generating a plurality of third grayscale images; 、 Based on multiple third grayscale images The second color image Generate The ISO sensitivity information is a map showing the ISO sensitivity for each pixel. It is characterized by: [Effects of the Invention]

[0007] According to the present invention, a high-resolution output image can be obtained in image processing using a machine learning model. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram of an image processing system according to a first embodiment. [Figure 2] 1 is an external view of an image processing system according to a first embodiment. [Figure 3] FIG. 2 is a conceptual diagram showing a method for learning weights of a neural network in the first embodiment. [Figure 4] 1 is a flowchart relating to learning of weights of a neural network in the first embodiment. [Figure 5] FIG. 2 is a conceptual diagram showing a method for generating an output image using a neural network in the first embodiment. [Figure 6] 1 is a flowchart relating to generation of an output image using a neural network in the first embodiment. [Figure 7] FIG. 10 is a block diagram of an image processing system according to a second embodiment. [Figure 8] FIG. 10 is an external view of an image processing system according to a second embodiment. [Figure 9] 10 is a flowchart relating to generation of an output image using a neural network in the second embodiment. [Figure 10] FIG. 10 is a block diagram of an image processing system according to a third embodiment. [Figure 11] 11 is a flowchart relating to generation of an output image using a neural network in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted.

[0010] First, before describing the specific embodiments, the gist of the present embodiment will be described. In this embodiment, a luminance image (grayscale image) is upscaled using a machine learning model. In this embodiment, image processing for enlarging and increasing the resolution of an image is referred to as upscaling. The machine learning model in this embodiment is generated by performing learning using a neural network. The neural network uses a filter that is convolved with an image, a bias that is added, and an activation function that performs nonlinear transformation. The filter and bias are called weights, and are learned (updated) using training images and ground truth images. In this embodiment, the machine learning model is learned using images expressed in grayscale as training images and ground truth images.

[0011] The image processing method of this embodiment includes a step of dividing a first grayscale image to generate a plurality of second grayscale images having a lower resolution than the first grayscale image, and an estimation step of inputting the plurality of second grayscale images to a machine learning model to generate a plurality of upscaled third grayscale images.

[0012] In this embodiment, the input image to the machine learning model is a grayscale image that has been reduced relative to the original grayscale image by reversibly dividing the grayscale image. When generating an input image from a grayscale image, the original grayscale image can be reduced reversibly by dividing it into multiple images. This reduces the loss of information that accompanies reduction, making it possible to obtain a highly accurate estimated image (output image). Furthermore, because the input image is a reduced grayscale image that has less information (number of channels) than a color image, the number of operations required for image processing can be reduced, which is another feature of this embodiment, as it enables faster image processing.

[0013] The above image processing method is an example, and the present invention is not limited to this. Details of other image processing methods will be described in the following examples.

[0014] [Example 1] First, an image processing system 100 according to a first embodiment of the present invention will be described. The image processing system 100 according to this embodiment uses a machine learning model to learn and execute image processing for upscaling an image.

[0015] Fig. 1 is a block diagram of an image processing system 100 in this embodiment. Fig. 2 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device 103, a display device 104, a recording medium 105, an input device 106, an output device 107, and a network 108.

[0016] The learning device 101 includes a storage unit (storage means) 101a, an acquisition unit (acquisition means) 101b, a generation unit (generation means) 101c, a division unit (division means) 101d, and a learning unit (learning means) 101e.

[0017] The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives an optical image of the subject formed via the optical system 102a to obtain the captured image 20. The imaging element 102b is a CCD (Charge Coupled Device) sensor, a CMOS (Complementary Metal-Oxide Semiconductor) sensor, or the like.

[0018] The imaging device 102 transmits the obtained image to an acquisition unit 103b of an image estimation device (image processing device) 103 (described later). If necessary, the imaging device 102 may transmit the shooting conditions corresponding to the captured image 20 along with the captured image 20. The shooting conditions are the conditions for capturing images when the captured image 20 is acquired using the optical system 102a and the image sensor 102b. Examples of the shooting conditions include the pixel pitch of the image sensor 102b, the type of optical low-pass filter of the optical system 102a, and ISO sensitivity. The shooting conditions may also be the conditions for developing the captured image 20 from an undeveloped RAW image in the imaging device 102. Examples of the conditions include noise reduction intensity, sharpness intensity, and image compression rate. In this embodiment, development refers to the process of converting a RAW image into an image file such as JPEG (Joint Photographic Experts Group) or TIFF (Tag Image File Format).

[0019] It should be noted that the memory unit that stores the images acquired by the imaging device 102, the display unit that displays the images, the transmission unit that transmits the images to the outside, the output unit that stores the images in an external storage medium, and the control unit that controls each part of the imaging device 102 are not shown in the figure.

[0020] The image estimation device 103 has a memory unit (storage means) 103a, an acquisition unit (acquisition means) 103b, a generation unit (generation means) 103c, a division unit (division means) 103d, and a processing unit (estimation means) 103e, and performs image processing on the acquired captured image 20 to generate an output image.

[0021] The acquiring unit 103b acquires the captured image 20. Note that the acquiring unit 103b may acquire (receive) the photographing conditions corresponding to the captured image 20 together with the captured image 20, as necessary.

[0022] The generation unit 103c extracts a Y image (luminance image) and multiple color difference images (first color difference images) by YUV converting the acquired captured image 20. The luminance image is a grayscale image that represents luminance value information only by shades of a single color. The color difference images are images that have information on U and V after YUV conversion. Details of the YUV conversion will be described later.

[0023] The dividing unit 103d reduces the luminance image by dividing (transforming) the obtained luminance image.

[0024] The processing unit 103e generates an estimated image (output image) by performing image processing to enlarge and increase the resolution of the reduced luminance image (input image). The processing unit 103e may perform image processing using the shooting conditions acquired by the acquisition unit 103b. For example, by using the pixel pitch of the image sensor, the type of optical low-pass filter, and the image compression rate in addition to the input image when training a machine learning model, image processing can be performed even on images acquired by any image capture device in which the ground truth images corresponding to the training images are different. Details of image processing using the shooting conditions will be described later. The captured image 20 may be an image captured by the image capture device 102 or an image stored on the recording medium 105. Furthermore, the captured image 20 may be an image that is originally expressed in grayscale, such as an infrared image or a distance image.

[0025] The image processing in this embodiment uses a neural network. Weight information in the neural network is learned by the learning device 101. The image estimation device 103 reads out the weight information from the storage unit 101a via the network 108 and stores it in the storage unit 103a. The stored weight information may be the weight numerical value itself or in an encoded format. Details regarding weight learning and image processing using the weight will be described later. The image estimation device 103 has the function of performing development processing and other image processing as necessary.

[0026] The output image is output to at least one of a display device 104, a recording medium 105, and an output device 107. The display device 104 is, for example, a liquid crystal display or a projector. A user can check the image being processed via the display device 104 and perform image editing work via the input device 106. The recording medium 105 is, for example, a semiconductor memory, a hard disk, a server on a network, etc. The input device 106 is, for example, a keyboard or a mouse, etc. The output device 107 is, for example, a printer, etc. The image estimation device 103 may also display or output the image that has been subjected to colorization processing. The colorization processing will be described later.

[0027] Next, a weight learning method (a method for manufacturing a trained model) executed by the learning device 101 in this embodiment will be described with reference to Figs. 3 and 4. Fig. 3 is a conceptual diagram showing learning (updating) of weights of a neural network. Fig. 4 is a flowchart related to neural network learning. In this embodiment, a convolutional neural network (CNN) 30 is used as the neural network. However, this embodiment is not limited to this, and a generative adversarial network (GAN) or a recurrent neural network (RNN) may also be used, for example.

[0028] In this embodiment, the weights of the CNN 30 are learned (updated) using mini-batch learning. In mini-batch learning, the errors between a plurality of ground truth images and their corresponding estimated images are calculated, and the weights are updated. For example, the L2 norm or the L1 norm can be used as the loss function. However, the embodiment is not limited to this, and online learning or batch learning may also be used.

[0029] The convolutional layer CN performs a filter convolution operation on the information input to the CNN 30, and calculates the sum of the input information and a bias. Furthermore, the convolutional layer CN performs a nonlinear transformation on the result of the operation based on an activation function. The initial values of each filter component and bias are arbitrary and are determined by random numbers in this embodiment. The activation function may be, for example, a ReLU (Rectified Linear Unit) or a sigmoid function. Each convolutional layer CN, except for the final layer, outputs a feature map. In this embodiment, the feature map is a four-dimensional array, with dimensions of batch, vertical, horizontal, and channel.

[0030] The skip connection SC combines feature maps output from discontinuous layers. In this embodiment, the feature maps are combined using an element-wise summation method. Note that the feature maps may also be combined by channel-wise concatenation.

[0031] Pixel Shuffle (PS) is a method for enlarging a feature map. In this embodiment, a low-resolution feature map is enlarged in a layer close to the output layer to create a high-resolution feature map. Note that the feature map may be enlarged using, for example, deconvolution or transposed convolution.

[0032] A residual block RB (Residual Block) is an element (block or module) that combines multiple convolution layers CN. To achieve more accurate learning, learning may be performed using a network in which residual blocks are layered, called a residual network. In this embodiment, a residual network is used as the layered network, but this is not limiting. For example, a network may be configured by layering elements such as an inception module and a dense block.

[0033] If necessary, in the convolutional layer CN, the feature map may be reduced in layers close to the input layer and expanded in layers close to the output layer, thereby reducing the size of the feature map in the intermediate layer to reduce the processing load. Here, pooling or stride can be used to reduce the feature map. Furthermore, deconvolution, pixel shuffling, interpolation, etc. can be used to expand the feature map.

[0034] Next, a flowchart relating to neural network learning will be described. The steps in Fig. 4 are mainly executed by the acquisition unit 101b, the generation unit 101c, the division unit 101d, and the learning unit 101e.

[0035] First, in step S101 (acquisition step), the acquisition unit 101b acquires a first supervised patch 10 (first supervised image) and a first training patch 11 (first training image). The first supervised patch 10 and the first training patch 11 are grayscale images containing at least brightness information. In this embodiment, the first supervised patch 10 has a larger image size and higher resolution than the first training patch 11, and depicts the same subject as the first training patch 11. A patch is an image having a predetermined number of pixels. For example, the first training patch 11 has 128 × 128 × 1 pixels, and the corresponding first supervised patch 10 has 256 × 256 × 1 pixels. Note that the batch magnification is not limited to 2 times in both the vertical and horizontal directions, and may be any number of times as long as the first training patch 11 and the corresponding first supervised patch 10 can be acquired. In this embodiment, the first training patch 11 and the corresponding first answer patch 10 are generated by numerical calculation, but the present invention is not limited to this. For example, the first training patch 11 and the corresponding first answer patch 10 may be obtained by capturing images of the same subject using optical systems with different focal lengths and cropping corresponding portions of the two resulting images. Alternatively, the first training patch 11 may be generated by reducing the resolution of the first answer patch 10 through downsampling. Furthermore, the first answer patch 10 and the first training patch 11 may each be a luminance patch (grayscale image) obtained by YUV converting a color patch. By YUV converting the color patch, a luminance patch and multiple chrominance patches can be generated. The luminance patch and multiple chrominance patches are generated from the color patch according to the following equation. However, this embodiment is not limited to this, and other definition equations may be used. [Number 1] Y=0.299R+0.587G+0.114B U=-0.14713R-0.28886G+0.436B V=0.615R-0.54199G-0.10001B

[0036] The above formula is used to convert from the RGB color space to the YUV color space. The RGB color space is represented using three color channels: Red, Green, and Blue. On the other hand, the YUV color space is represented using a luminance channel (Y) and two color difference channels (U and V).

[0037] In this embodiment, the acquisition unit 101b acquires the first supervised patch 10 and the first training patch 11 expressed in grayscale, but the acquisition unit 101b may acquire training color patches and supervised color patches having multiple color channels. In this case, the generation unit 101c generates the first supervised patch 10 and the first training patch 11 from the supervised color patch and the training color patch according to Equation 1. Note that only one of the first supervised patch 10 and the first training patch 11 may be generated from a color patch, and the other may be acquired by the acquisition unit 101b as a luminance patch.

[0038] Next, in step S102 (division step), the division unit 101d divides the first training patch 11 to generate multiple second training patches 12 (second training images). The multiple second training patches 12 are generated by reversible transformation that does not lose information during division. In this embodiment, the second training patches 12 are generated by spatially arranging pixel values extracted from every other pixel in the vertical direction and every other pixel in the horizontal direction of the first training patch 11. In this case, four second training patches 12 can be generated in the channel (depth) direction from one first training patch 11. Furthermore, each second training patch 12 has at least one smaller vertical and horizontal size and lower resolution than the first training patch 11. Furthermore, because the second training patches 12 are reversibly transformed, the sum of the number of pixels in the multiple second training patches 12 is equal to the number of pixels in the first training patch 11.

[0039] In this embodiment, the first training patch 11 is divided into four equal second training patches 12 each having the same number of pixels, but this is not limiting, and it is sufficient that at least the first training patch 11 and the plurality of second training patches 12 are transformed reversibly. For example, the plurality of second training patches 12 may each have a different number of pixels, and any number of second training patches 12 greater than or equal to two may be generated instead of four. Furthermore, frequency components obtained by multi-resolution analysis using discrete wavelet transform may be used.

[0040] In this way, by reversibly transforming the first training patch 11 expressed in grayscale and using the multiple second training patches 12 reduced in the spatial direction as input images to the CNN 30, it is possible to reduce the amount of calculation in the CNN 30. Furthermore, since the multiple second training patches 12 do not lose information when the first training patch 11 is divided, it is possible to perform image processing with high accuracy.

[0041] In step S102, the dividing unit 101d generates a plurality of second supervised patches 14 (second supervised images) by dividing the first supervised patch 10 in the same manner as the first training patch 11. In addition, the dividing unit 101d inputs the shooting conditions to the CNN 30 along with the plurality of second training patches 12, and therefore may convert the second training patches 12 into an image (map) having the shooting conditions for each pixel based on the shooting conditions acquired by the acquiring unit 103b.

[0042] Next, in step S103 (estimation step), the learning unit 101e uses a CNN 30 (machine learning model) to perform image processing on the divided second training patches 12, thereby generating a plurality of estimated patches 13 (estimated images). The plurality of estimated patches 13 are estimated images obtained by the CNN 30, and ideally, each corresponds to a corresponding plurality of second ground truth patches 14. The learning unit 101e can also input the shooting conditions to the CNN 30 by concatenating an image having a shooting condition for each pixel in the channel direction of the plurality of second training patches 12. When an image having a shooting condition for each pixel is input to the CNN 30 together with the second training patches 12, the learning unit 101e performs image processing based on the shooting conditions in addition to upscaling, thereby generating a plurality of estimated patches 13.

[0043] Subsequently, in step S104 (update step), the learning unit 101e updates the weights of the CNN 30 based on the error (loss) between the estimated patch 13 and the second correct answer patch 14. Here, the weights include the filter components and biases of each layer. In this embodiment, backpropagation is used to update the weights, but the present invention is not limited to this. For example, gradient descent may also be used.

[0044] Next, in step S105, the learning unit 101e determines whether weight learning is complete. Completion can be determined by, for example, whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight during update is smaller than a specified value. If it is determined that learning is incomplete, the process returns to step S101, and a new first training patch 11 and a corresponding first correct answer patch 10 are obtained. On the other hand, if it is determined that learning is complete, the learning device 101 ends learning and stores weight information in the storage unit 101a.

[0045] Next, the generation of an output image in this embodiment will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a conceptual diagram showing the generation of an output image of a neural network. Fig. 6 is a flowchart related to the generation of an output image using a neural network. The steps in Fig. 6 are mainly executed by the acquisition unit 103b, generation unit 103c, division unit 103d, and processing unit 103e of the image estimation device (image processing device) 103.

[0046] First, in step S201 (acquisition step), the acquisition unit 103b acquires a captured image 20 (first color image). The captured image 20 is an image that includes at least luminance information, as in learning. In this embodiment, the captured image 20 is a color image transmitted from the imaging device 102, but the present invention is not limited to this. For example, the captured image 20 may be an image stored in the storage unit 103a, or may be a grayscale image that represents only luminance information using shades of a single color. Note that the shooting conditions corresponding to the captured image 20 may be acquired together with the captured image 20 and used in subsequent steps.

[0047] Next, in step S202 (generation step), the generation unit 103c performs YUV conversion on the acquired captured image 20 to extract a Y image (luminance image) and multiple color difference images (first color difference images). The luminance image is a first grayscale image 21 that represents only the luminance information of the captured image 20 using only shades of a single color. The multiple color difference images are multiple color difference images 22 (first color difference images) that have information about the color differences of the captured image 20. The Y image and multiple color difference images can be generated from the captured image 20 according to Equation 1.

[0048] Next, in step S203 (division step), the division unit 103d divides the first grayscale image 21 into a plurality of second grayscale images 23. At this time, the plurality of second grayscale images 23 are generated by reversible division, which does not cause loss of information during division. Therefore, each second grayscale image 23 is smaller in at least one of the vertical and horizontal dimensions and has a lower resolution than the first grayscale image 21. Furthermore, since the second grayscale images 23 are reversibly transformed, the sum of the number of pixels of the second grayscale images 23 is equal to the number of pixels of the first grayscale image 21. Furthermore, it is preferable that the plurality of second grayscale images have the same number of pixels (resolution). When the plurality of second grayscale images have the same number of pixels, the amount of calculation for each of the plurality of second grayscale images is the same, which can improve the efficiency of calculation in the estimation step described below. Note that the method for dividing the first grayscale image 21 is similar to the method for transforming the first training patch 11 in step S102, and therefore a description thereof will be omitted.

[0049] Subsequently, in step S204 (estimation step), the processing unit 103e performs image processing using the CNN 30 to generate a plurality of first estimated images 24 (third grayscale images) from the plurality of second grayscale images 23. Note that the weight information used to generate the plurality of first estimated images 24 is transmitted from the learning device 101 and stored in the storage unit 103a, and is the same neural network as in FIG.

[0050] If necessary, in step S205 (combining step), the processing unit 103e may further perform image processing on the multiple first estimated images 24. For example, a second estimated image 25 (fourth grayscale image) can be generated by combining (synthesizing) the multiple first estimated images 24. At this time, the second estimated image 25 is generated from the multiple first estimated images 24 by performing the reverse operation of the method used to transform the first grayscale image 21 into the multiple second grayscale images 23 in step S203. That is, in this embodiment, the processing unit 103e can generate the second estimated image 25 by adding the multiple first estimated images 24 together in the spatial direction. At this time, the number of pixels in the second estimated image 25 is equal to the sum of the number of pixels in the multiple first estimated images 24. Therefore, by using multiple second grayscale images 23 obtained by reducing the first grayscale image 21 as input images to the machine learning model, the amount of calculation can be reduced compared to using a color image as the input image when upscaling at the same magnification. When the second estimated image 25 is generated by combining a plurality of first estimated images 24, the image estimation device 103 may use the second estimated image 25 as the output image.

[0051] Furthermore, in step S206 (colorization step), the processing unit 103e may perform image processing to colorize the second estimated image 25. At this time, colorization is performed based on the second estimated image 25 and the multiple color difference images 22 generated in step S202, to generate an estimated color image 26 (second color image). The estimated color image 26 is an image obtained by upscaling the captured image 20. In this embodiment, the colorization of the luminance image is performed according to Equation 2. [Number 2] R=Y+1.13983V G=Y-0.39465U-0.58060V B=Y+2.03211U

[0052] The above formula is used for conversion from YUV color space to RGB color space. Formula 2 is the inverse operation of the conversion from RGB color space to YUV color space performed according to Formula 1. If another definition formula is used as a method for generating a luminance image from a color image, the inverse operation must be used as a method for generating a color image from a luminance image. If estimated color image 26 is generated by colorizing second estimated image 25, image estimation device 103 may use estimated color image 26 as an output image.

[0053] Furthermore, the processing unit 103e may use multiple color difference interpolated images 27 (second color difference images) to generate the estimated color image 26. The multiple color difference interpolated images 27 are generated by interpolating each of the multiple color difference images 22 to achieve higher resolution (interpolation step). Note that the method for generating the color difference interpolated image 27 from the color difference image 22 is not limited to this, and may be performed using, for example, a binary method or a bicubic method, or a method using a machine learning model. In this case, it is preferable that each of the multiple color difference interpolated images 27 has the same resolution (number of pixels) as the second estimated image 25. Colorizing the second estimated image 25 using multiple color difference interpolated images 27 having the same resolution as the second estimated image 25 reduces noise due to colorization, resulting in a more accurate estimated color image 26.

[0054] In this embodiment, a method has been described in which a captured image 20 is acquired in step S201, and a first grayscale image 21 is generated from the captured image 20 in step S202 to generate an output image in steps S203 and after. However, if the acquisition unit 103b acquires an image that is initially expressed in grayscale (for example, an infrared image or a distance image) in step S201, steps S201 and S202 can be skipped and steps S203 and after can be executed. In this case, since there is no information regarding the color difference of the captured image 20, the second estimated image 25 cannot be colorized.

[0055] In this embodiment, the learning device 101 and the image estimation device 103 are separate devices, but the present invention is not limited to this. The learning device 101 and the image estimation device 103 may be integrated. In other words, the learning process and the estimation process may be performed within a single device.

[0056] With the above configuration, according to this embodiment, an image processing system can be provided that obtains a high-resolution output image by using a grayscale image reduced by reversible division as an input image in image processing using a machine learning model.

[0057] [Example 2] Next, an image processing system 200 according to a second embodiment of the present invention will be described. The image processing system 200 according to this embodiment uses a machine learning model to learn and execute image processing for upscaling an image.

[0058] The image processing system 200 of this embodiment differs from the first embodiment in that an image capturing device 202 acquires a captured image 20 and processes the image.

[0059] Fig. 7 is a block diagram of an image processing system 200 in this embodiment. Fig. 8 is an external view of the image processing system 200. The image processing system 200 has a learning device 201 and an imaging device 202 connected via a network 203. Note that the learning device 201 and the imaging device 202 do not need to be constantly connected via the network 203.

[0060] The learning device 201 has a memory unit (storage means) 211, an acquisition unit (acquisition means) 212, a generation unit (generation means) 213, a division unit (division means) 214, and a learning unit (learning means) 215. These are used to learn (update) the weights of the neural network in order to upscale the captured image 20. Note that information about the weights of the neural network is learned in advance by the learning device 201 and stored in the memory unit 211. The method of learning (updating) the weights of the neural network executed by the learning device 201 is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0061] The imaging device 202 has an optical system 221, an imaging element 222, an image estimation unit 223, a storage unit 224, a recording medium 225a, a display unit 225b, an input unit 226, and a system controller 227. The imaging device 202 captures an image of a subject space to obtain a captured image 20 and generates an output image. The optical system 221 and the imaging element 222 in the imaging device 202 are the same as those in the first embodiment, and therefore a description thereof will be omitted. The imaging device 202 also reads out weight information from the storage unit 211 via the network 203 and stores the weight information in the storage unit 224.

[0062] The image estimation unit 223 has an acquisition unit 223a, a generation unit 223b, a division unit 223c, and a processing unit 223d. The acquisition unit 223a acquires a captured image 20 and shooting conditions corresponding to the captured image 20 from the imaging device 202. The generation unit 223b and the division unit 223c are similar to the generation unit 103c and the division unit 103d in the first embodiment. The captured image 20 acquired by the acquisition unit 223a is subjected to image processing based on weight information stored in the storage unit 224, and an output image is generated. In this embodiment, the processing unit 223d uses shooting conditions corresponding to the captured image 20 for image processing.

[0063] The recording medium 225a stores the output image. When a user issues an instruction to display an estimated image via the input unit 226, the stored output image is read out and displayed on the display unit 225b. The image estimation unit 223 may read out the captured image 20 and the shooting conditions stored in the recording medium 225a and perform processing to generate the output image. The system controller 227 controls the processing performed by the imaging device 202.

[0064] Next, the generation of an output image in this embodiment will be described. Fig. 9 is a flowchart related to the generation of an output image using a neural network in this embodiment. The steps in the generation of second estimated image 25 are mainly executed by acquisition unit 223a (acquisition means), generation unit (generation means) 223b, division unit (division means) 223c, and processing unit (estimation means) 223d of image estimation unit 223.

[0065] First, in step S301 (acquisition step), the acquisition unit 223a acquires the captured image 20 and the shooting conditions corresponding to the captured image 20. In this embodiment, the captured image 20 is a color image that is acquired by the imaging device 202 and stored in the storage unit 224. Step S302 (generation step) and step S303 (division step) are similar to steps S202 and S203 in the first embodiment, and therefore their explanation will be omitted.

[0066] Next, in step S304 (estimation step), the processing unit 223d performs image processing using a neural network to generate multiple estimated images (third grayscale images) 24 from the multiple second grayscale images 23. Note that the weight information used to generate the estimated images is transmitted from the learning device 101 and stored in the storage unit 103a, and is the same neural network as in FIG. 3. In this embodiment, the processing unit 223d performs image processing using ISO sensitivity as a shooting condition in addition to the multiple first estimated images 24. ISO sensitivity is a shooting condition that indicates how sensitive a sensor is to light, and a high ISO sensitivity makes it more likely that noise will appear in the image. By using ISO sensitivity as a shooting condition, image processing can be performed so as not to overemphasize noise when upscaling a captured image 20 with a high ISO sensitivity.

[0067] Note that the shooting conditions are not limited to ISO sensitivity, and for example, noise removal strength may be used as a shooting condition, and if the noise removal strength of the captured image 20 is weak (the captured image 20 contains many high-frequency components), image processing is performed to reduce the high-frequency components of the output image. Also, sharpness strength may be used as a shooting condition, and if the sharpness strength of the captured image 20 is strong (the captured image 20 contains many high-frequency components), image processing is performed to prevent the output image from containing excessive high-frequency components. Furthermore, image compression rate may be used as a shooting condition, and if the image compression rate of the captured image 20 is high (the captured image 20 contains many high-frequency components), image processing is performed to compensate for the high-frequency components of the output image.

[0068] Subsequently, in step S305 (processing step), the processing unit 223d generates an output image by combining and colorizing the multiple first estimated images 24. Note that the combining and colorizing methods are the same as those in the first embodiment, and therefore will not be described here.

[0069] With the above configuration, this embodiment can provide an image processing system that obtains a high-resolution output image by using a grayscale image that has been reduced by reversible deformation as an input image in image processing using a machine learning model. Note that in this embodiment, by inputting the shooting conditions together with the reduced grayscale image into the machine learning model, image processing can be performed with higher accuracy.

[0070] [Example 3] Next, an image processing system 300 according to a third embodiment of the present invention will be described. The image processing system 300 according to this embodiment learns and executes image processing for upscaling an image using a machine learning model.

[0071] The image processing system 300 of this embodiment differs from the first embodiment in that it includes a control device 304 that acquires a captured image 20 from an imaging device 302 and issues a request for image processing of the captured image 20 to an image estimation device (image processing device) 303.

[0072] 10 is a block diagram of an image processing system 300 in this embodiment. The image processing system 300 includes a learning device 301, an imaging device 302, an image estimation device 303, and a control device 304. In this embodiment, the learning device 301 and the image estimation device 303 may be servers. The control device 304 is a user terminal such as a personal computer or a smartphone. The control device 304 is connected to the image estimation device 303 via a network 305. The image estimation device 303 is connected to the learning device 301 via a network 306. In other words, the control device 304 and the image estimation device 303, and the image estimation device 303 and the learning device 301 are configured to be able to communicate with each other.

[0073] The learning device 301 and the imaging device 302 in the image processing system 300 have the same configurations as the learning device 101 and the imaging device 102, respectively, and therefore their explanations will be omitted.

[0074] Image estimation device 303 includes storage unit 303a, acquisition unit (acquisition means) 303b, generation unit (generation means) 303c, division unit (division means) 303d, processing unit (estimation means) 303e, and communication unit (reception means) 303f. Storage unit 303a, acquisition unit 303b, generation unit 303c, division unit 303d, and processing unit 303e in image estimation device 303 are similar to storage unit 103a, acquisition unit 103b, generation unit 103c, division unit 103d, and processing unit 103e, respectively.

[0075] The control device 304 includes a communication unit (transmission means) 304a, a display unit (display means) 304b, an input unit (input means) 304c, a processing unit (processing means) 304d, and a recording unit 304e. The communication unit 304a can transmit a request to the image estimation device 303 to cause the image estimation device 303 to execute processing on the captured image 20. The communication unit 304a can also receive an output image processed by the image estimation device 303. The communication unit 304a may communicate with the imaging device 302. The display unit 304b displays various information. The various information displayed by the display unit 304b includes, for example, the captured image 20 to be transmitted to the image estimation device 303 or the output image received from the image estimation device 303. The input unit 304c allows a user to input instructions to start image processing, etc. The processing unit 304d can perform image processing, including colorization, on the output image received from the image estimation device 303. The recording unit 304 e stores the captured image 20 acquired from the imaging device 302 and the output image received from the image estimation device 303 .

[0076] The method of transmitting the captured image 20 to be processed to the image estimation device 303 is not limited. For example, the captured image 20 may be uploaded to the image estimation device 303 simultaneously with S401, or may be uploaded to the image estimation device 403 before S401. Furthermore, the captured image 20 may be an image stored on a server different from the image estimation device 303.

[0077] Next, the generation of an output image in this embodiment will be described. Fig. 11 is a flowchart showing the generation of an output image using a neural network in this embodiment.

[0078] The following describes the operation of the control device 304. In this embodiment, image processing is started by a user issuing an instruction to start image processing via the control device 304.

[0079] First, in step S401 (first transmission step), the communication unit 304a transmits a request for processing the captured image 20 to the image estimation device 303. Note that in step S401, the control device 304 may transmit an ID for authenticating a user, shooting conditions corresponding to the captured image 20, and the like, together with the request for processing the captured image 20.

[0080] Subsequently, in step S402 (first receiving step), the communication unit 304a receives the output image generated by the estimation device 303.

[0081] Next, a description will be given of the operation of the image estimation device 403. First, in step S501, the communication unit 303f receives a request for processing the captured image 20 transmitted from the communication unit 304a. Upon receiving the instruction to process the captured image 20, the image estimation device 303 executes the processing from step S502 onwards.

[0082] Next, in step S502, the acquisition unit 303b acquires the captured image 20. In this embodiment, the captured image 20 is transmitted from the control device 304. At this time, the photographing conditions corresponding to the captured image 20 may be acquired together with the captured image 20. Note that the processes of steps S501 and S502 may be performed simultaneously. Steps S503 to S505 are similar to steps S202 to S204, and therefore description thereof will be omitted.

[0083] Subsequently, in step S506, the image estimation device 303 transmits the output image to the control device 304. The output image transmitted by the image estimation device 303 includes any one of the plurality of first estimated images 24, the second estimated image 25 generated from the plurality of first estimated images 24, and the estimated color image 26.

[0084] With the above configuration, this embodiment can provide an image processing system that obtains a high-resolution output image by using a grayscale image that has been reduced by reversible deformation in image processing using a machine learning model as an input image. In this embodiment, the control device 304 only requests processing for a specific image. Actual image processing is performed by the image estimation device 303. Therefore, if the control device 304 is a user terminal, the processing load on the user terminal can be reduced. Therefore, the user can obtain an output image with a low processing load.

[0085] [Other Examples] This embodiment can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0086] According to each embodiment, it is possible to provide an image processing method, an image processing device, a program, and a storage medium that are capable of obtaining a high-resolution output image in image processing using a machine learning model. The image processing device may be any device that has the image processing function of the embodiments, and may be realized in the form of an imaging device or a personal computer.

[0087] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0088] [Method 1] generating a plurality of second grayscale images having a lower resolution than the first grayscale image by dividing the first grayscale image; and generating a plurality of upscaled third grayscale images by inputting the plurality of second grayscale images into a machine learning model.

[0089] [Method 2] The image processing method according to Method 1, wherein the resolutions of the second grayscale images are the same as each other.

[0090] [Method 3] 3. The image processing method of any one of methods 1 or 2, further comprising the step of combining the third grayscale images together to generate a fourth grayscale image.

[0091] [Method 4] The image processing method according to Method 3, wherein the number of pixels in the fourth grayscale image is equal to the sum of the number of pixels in the plurality of third grayscale images.

[0092] [Method 5] generating the first grayscale image and a plurality of first color difference images from a first color image; and generating a second color image based on the fourth grayscale image and the plurality of first color difference images.

[0093] [Method 6] generating a plurality of second color difference images by interpolating the plurality of first color difference images; 6. The image processing method of Method 5, further comprising generating the second color image based on the fourth grayscale image and the plurality of second color difference images.

[0094] [Method 7] 7. The image processing method of claim 6, wherein the resolution of each of the plurality of second color difference images is the same as the resolution of the fourth grayscale image.

[0095] [Method 8] the first grayscale image is acquired by photographing using an optical system and an image sensor; 5. The image processing method according to any one of Methods 1 to 4, wherein the step of generating the plurality of third grayscale images comprises inputting the plurality of second grayscale images and the shooting conditions of the shooting into a machine learning model to generate the plurality of upscaled third grayscale images.

[0096] [Method 9] the first color image is acquired by photographing using an optical system and an image sensor; 8. The image processing method according to any one of Methods 5 to 7, wherein the step of generating the plurality of third grayscale images comprises inputting the plurality of second grayscale images and the shooting conditions of the shooting into a machine learning model to generate the plurality of upscaled third grayscale images.

[0097] [Method 10] 10. The image processing method according to any one of Methods 8 and 9, wherein the shooting conditions include at least one of the pixel pitch of the image sensor, the type of optical low-pass filter of the optical system, and ISO sensitivity.

[0098] [Method 11] 10. The image processing method according to any one of Methods 8 and 9, wherein the shooting conditions include at least one of noise reduction intensity, sharpness intensity, and image compression rate.

[0099] [Program 12] 12. A program for causing a computer to execute the image processing method according to any one of Methods 1 to 11.

[0100] [Configuration 13] A storage medium storing the program described in Program 12.

[0101] [Configuration 14] a division unit that divides a first grayscale image to generate a plurality of second grayscale images having a lower resolution than the first grayscale image; and a processing unit that generates a plurality of upscaled third grayscale images by inputting the plurality of second grayscale images into a machine learning model.

[0102] [Configuration 15] an acquisition unit that acquires a first training image and a first ground truth image; a division unit that divides the first training images and the first gold standard image to generate a plurality of second training images having a lower resolution than the first training images and a plurality of second gold standard images having a lower resolution than the first gold standard image; a processor that generates a plurality of upscaled estimated images by inputting the plurality of second training images into a machine learning model; a learning unit that updates weights of a neural network based on the plurality of estimated images and the plurality of second correct images.

[0103] [Method 16] obtaining a first training image and a first ground truth image; generating a plurality of second training images having a lower resolution than the first training images and a plurality of second ground truth images having a lower resolution than the first ground truth images by dividing the first training images and the first ground truth images; generating a plurality of upscaled estimated images by inputting the plurality of second training images into a machine learning model; and updating weights of a neural network based on the plurality of estimated images and the plurality of second ground truth images.

[0104] [Program 17] A program that causes a computer to execute the method for producing a trained model described in Method 16.

[0105] [Configuration 18] An image processing system including an imaging device and a learning device capable of communicating with the imaging device, The learning device includes an acquisition unit that acquires a first training image and a first correct answer image; a division unit that divides the first training images and the first gold standard image to generate a plurality of second training images having a lower resolution than the first training images and a plurality of second gold standard images having a lower resolution than the first gold standard image; a processor that generates a plurality of upscaled estimated images by inputting the plurality of second training images into a machine learning model; a learning unit that updates weights of a neural network based on the plurality of estimated images and the plurality of second ground truth images, the imaging device includes an optical system, an imaging element, and an image estimation unit; The image estimation unit includes: a means for acquiring a first grayscale image; means for dividing a first grayscale image to generate a plurality of second grayscale images having a lower resolution than the first grayscale image; an image processing system comprising: means for generating a plurality of third grayscale images by upscaling the plurality of second grayscale images based on the plurality of second grayscale images.

[0106] [Configuration 19] An image processing system including a control device and an image processing device capable of communicating with the control device, the control device has a means for transmitting a request to the image processing device to execute processing on the captured image, The image processing device includes: means for receiving the request; means for acquiring the captured image; means for generating a plurality of first grayscale images from the captured image; and means for generating a plurality of upscaled second grayscale images by inputting the plurality of first grayscale images into a machine learning model. [Explanation of symbols]

[0107] 21 First Grayscale Image 23 Second Grayscale Image 24 Third Grayscale Image

Claims

1. 1. An image processing method for generating, based on a first color image acquired by photography, a second color image having a larger number of pixels than the first color image, comprising: generating a first grayscale image from the first color image; generating a plurality of second grayscale images each having a smaller number of pixels than the first grayscale image by dividing the first grayscale image; generating a plurality of third grayscale images that are upscaled relative to the plurality of second grayscale images by channel-wise concatenating the plurality of second grayscale images and information on the ISO sensitivity at the time of shooting and inputting the concatenated data into a machine learning model; generating the second color image based on the plurality of third grayscale images; The image processing method, wherein the information on ISO sensitivity is a map showing ISO sensitivity for each pixel.

2. 2. The image processing method according to claim 1, wherein the number of pixels of each of the plurality of second grayscale images is the same as each other.

3. 2. The image processing method of claim 1, further comprising the step of generating a fourth grayscale image by combining the third grayscale images together.

4. 4. The image processing method according to claim 3, wherein the number of pixels in the fourth grayscale image is equal to the total number of pixels in the plurality of third grayscale images.

5. generating the first grayscale image and a plurality of first color difference images from the first color image; 4. The image processing method according to claim 3, wherein the second color image is generated based on the fourth grayscale image and the plurality of first color difference images.

6. generating a plurality of second color difference images by upscaling the plurality of first color difference images; 6. The image processing method according to claim 5, wherein the second color image is generated based on the fourth grayscale image and the plurality of second color difference images.

7. 7. The image processing method according to claim 6, wherein the number of pixels in each of the plurality of second color difference images is the same as the number of pixels in the fourth grayscale image.

8. 8. The image processing method according to claim 1, wherein the first grayscale image is a luminance image.

9. the first color image is acquired by photographing using an optical system and an image sensor; 8. The image processing method according to claim 1, wherein in the step of generating the plurality of third grayscale images, the plurality of second grayscale images, the ISO sensitivity, and shooting conditions for the shooting are input into a machine learning model to generate the plurality of third grayscale images.

10. 10. The image processing method according to claim 9, wherein the photographing conditions include at least one of a pixel pitch of the image sensor and a type of an optical low-pass filter of the optical system.

11. 10. The image processing method according to claim 9, wherein the photographing conditions include at least one of noise reduction intensity, sharpness intensity, and image compression rate.

12. 8. The image processing method according to claim 1, wherein the first grayscale image is a color difference image.

13. A program causing a computer to execute the image processing method according to any one of claims 1 to 7.

14. A storage medium storing the program according to claim 13.

15. 1. An image processing device that generates, based on a first color image acquired by photographing, a second color image having a larger number of pixels than the first color image, means for generating a first grayscale image from the first color image; means for dividing the first grayscale image to generate a plurality of second grayscale images each having a smaller number of pixels than the first grayscale image; a means for generating a plurality of third grayscale images that are upscaled relative to the plurality of second grayscale images by linking the plurality of second grayscale images and information on the ISO sensitivity at the time of shooting in a channel direction and inputting the linked information into a machine learning model; means for generating the second color image based on the plurality of third grayscale images; 10. An image processing apparatus, wherein the information on ISO sensitivity is a map showing ISO sensitivity for each pixel.

16. an acquisition unit that acquires a first training image and a first ground truth image; a division unit that divides the first training images and the first gold standard image to generate a plurality of second training images each having a smaller number of pixels than the first training images and a plurality of second gold standard images each having a smaller number of pixels than the first gold standard image; a processing unit that generates a plurality of estimated images that are upscaled for the plurality of second training images by channel-wise concatenating the plurality of second training images and information on ISO sensitivity corresponding to the first training images and inputting the concatenated images into a machine learning model; a learning unit that updates weights of a machine learning model based on the plurality of estimated images and the plurality of second correct images, The learning device is characterized in that the information on ISO sensitivity is a map showing ISO sensitivity for each pixel.

17. obtaining a first training image and a first ground truth image; generating a plurality of second training images each having a smaller number of pixels than the first training images and a plurality of second ground truth images each having a smaller number of pixels than the first ground truth images by dividing the first training images and the first ground truth images; generating a plurality of estimated images upscaled for the plurality of second training images by channel-wise concatenating the plurality of second training images and information on ISO sensitivity corresponding to the first training images and inputting the concatenated images into a machine learning model; updating weights of a machine learning model based on the plurality of estimated images and the plurality of second ground truth images; A method for generating a trained model, characterized in that the ISO sensitivity information is a map indicating the ISO sensitivity for each pixel.

18. A program causing a computer to execute the trained model generation method according to claim 17.

19. An image processing system including the image processing device according to claim 15 and a control device capable of communicating with the image processing device, the control device has a means for transmitting a request to the image processing device to execute processing on the captured image, The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and a means for generating the second color image in response to the request.

Images