Image processing device, image forming system, image processing method and program

Abstract

To prevent a deterioration of colors occurring in demosaicing processing on an image to improve the quality of the colors of the image.SOLUTION: An image processing apparatus holds a model that learns an image based on the luminance of the image. The image processing apparatus acquires a first image used for the model's learning. When a first luminance of the first image exceeds a threshold, the image processing apparatus creates a second image having a luminance different from the first luminance. The image processing apparatus corrects a parameter for learning the image in the model based on the first luminance and the second luminance of the second image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image forming system, an image processing method, and a program. [Background technology]

[0002] Image sensors used in imaging devices such as digital cameras have color filters, each representing one of the three primary colors of RGB, attached to each pixel. This allows each pixel to receive light components of a specific wavelength. The RGB color filters arranged on each pixel can be arranged, for example, in a Bayer array. Images captured using a Bayer array are so-called mosaic images, each consisting of pixels with RGB color information. The development processing unit of the imaging device performs demosaic processing to obtain a color image from the mosaic image captured by the image sensor. Demosaic processing involves applying a linear filter to each RGB pixel scattered across the mosaic image. As a result, missing RGB color information in each RGB pixel is interpolated based on the RGB pixels surrounding it. Because linear interpolation methods for interpolating color information have low interpolation accuracy, numerous nonlinear interpolation methods have been proposed. However, various nonlinear interpolation methods have image regions where color information cannot be completely interpolated. As a result, demosaic processing has problems, such as false colors and artifacts.

[0003] Therefore, a data-driven interpolation method that applies deep learning technology to demosaic processing has been proposed. Non-Patent Document 1 discloses a method for training a CNN (Convolutional Neural Network)-based demosaic network (hereinafter referred to as CNN) using a low-noise teacher image. The method in Non-Patent Document 1 normalizes pixel values ​​of the teacher image so that they fall within an interval from a minimum value of 0 to a maximum value of 1. This interval is called the pixel value range. An image processing device uses a teacher image containing pixel values ​​normalized to fall within the pixel value range to have the CNN learn the features of the teacher image. The image processing device inputs a mosaic image (input image) to a CNN that has already learned the teacher image, and the CNN performs inference (output) to convert the mosaic image into an RGB image. Here, inference is also called estimation. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Michael Gharbi and 3 others, “Deep Joint Demosaicking and Denoising”, [online], 2016, Siggraph Asia 2016, [searched on February 25, 2021], Internet<URL:https: / / groups.csail.mit.edu / graphics / demosaicnet / > Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a problem that when the pixel values ​​of a particular pixel and a particular channel (RGB) of the mosaic image are close to 0 or 1, artifacts occur in the image inferred by the CNN.

[0006] An object of the present invention is to suppress color degradation that occurs during image demosaicing and improve the color quality of an image. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, an image processing apparatus according to an embodiment of the present invention has the following arrangement: The image processing device performs demosaic processing on a mosaic image of a predetermined color filter array. a holding means for holding the model; teaching An acquisition means for acquiring a master image; Mosaic image In the above model Enter and get so that the pixel values ​​of the output image are included in the pixel value range of the teacher image. of the teacher image an expansion means for expanding a pixel value range; a generating means for generating a student image, which is a mosaic image corresponding to the teacher image and has a pixel value range that is approximately the same as the pixel value range expanded by the expanding means; and a generating means for generating a student image, which is a mosaic image corresponding to the teacher image and has a pixel value range that is approximately the same as the pixel value range expanded by the expanding means, ... Correction means for correcting parameters and, Equipped with R . [Effects of the Invention]

[0008] According to the present invention, it is possible to suppress color degradation that occurs during image demosaicing and improve the color quality of the image. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an image forming system according to an embodiment of the present invention. [Figure 2] 1A to 1C are diagrams illustrating generation of an inference image by CNN according to the present embodiment. [Figure 3] 1A and 1B are diagrams for explaining the principle of artifact generation in the prior art. [Figure 4] FIG. 1 is a block diagram of an image processing apparatus according to a first embodiment and a second embodiment. [Figure 5] 4 is a flowchart of image conversion processing according to the first and second embodiments. [Figure 6] FIG. 10 is a block diagram of an image processing apparatus according to a third embodiment. [Figure 7] 10 is a flowchart of an image conversion process according to the third embodiment. [Figure 8] 5A to 5C are diagrams illustrating a generation process of a teacher image according to the embodiment. [Figure 9] FIG. 4 is a diagram illustrating a pixel value range according to the embodiment. [Figure 10] 5A to 5C are diagrams illustrating a generation process of an input image according to the present embodiment. [Figure 11] 4A to 4C are diagrams for explaining the processing of CNN according to the present embodiment. [Figure 12] FIG. 10 is a diagram illustrating the flow of learning of a CNN according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] (First embodiment) An image forming system 10 according to this embodiment will be described below.

[0012] FIG. 1 is a diagram showing an example of the configuration of an image forming system 10 according to the first embodiment. The image forming system 10 includes an image processing device 100, an operation unit 110, an external storage device 108, and a display device 109. The image processing device 100 includes a CPU 101, a RAM 102, a ROM 103, a storage unit 104, an input interface 105, an output interface 106, an imaging unit 111, and a GPU 112. The components of the image processing device 100 are connected to one another via a bus 107. The image processing device 100 is also connected to the external storage device 108 and the operation unit 110 via the input interface 105. The image processing device 100 is also connected to the external storage device 108 and the display device 109 via the output interface 106.

[0013] The CPU 101 is a processor representing a central processing unit, and executes computer programs stored in a RAM 102 and a ROM 103 (described later) to control the overall operation of the image forming system 10. Although the case where the CPU 101 controls the entire image forming system 10 has been described as an example, the entire image forming system 10 may be controlled by a plurality of hardware (not shown) sharing the control-related processing.

[0014] The RAM 102 is a main working memory and has a storage area for temporarily storing computer programs and data read from the storage unit 104 and data received from the outside via the input interface 105. The RAM 102 is also used as a storage area used by the CPU 101 when executing various processes and as a storage area used by the GPU 112 when executing image processing. The ROM 103 is a readable memory and has a storage area for storing setting parameters for each unit in the image forming system 10, a boot program, etc.

[0015] The storage unit 104 is a mass information storage device such as an HDD (hard disk drive). The storage unit 104 may be a storage device such as an optical disk drive or flash memory. The storage unit 104 stores an OS (operating system), computer programs and data for causing the CPU 101 to execute various processes. The storage unit 104 also holds temporary data (e.g., input or output image data and image conversion information by the GPU 112) generated by the processing of each component of the image forming system 10. The computer programs and data stored in the storage unit 104 are read as appropriate under the control of the CPU 101 and stored in the RAM 102.

[0016] The input interface 105 is a serial bus interface such as USB or IEEE 1394. The CPU 101 receives data, commands, etc. from an external device (not shown) via the input interface 105. In this embodiment, the CPU 101 obtains data from an external storage device 108 via the input interface 105.

[0017] The output interface 106 has the same configuration as the input interface 105 and is, for example, a serial bus interface such as USB or IEEE1394. The output interface 106 may also be, for example, a video output terminal such as DVI or HDMI (registered trademark). The image processing device 100 outputs data, etc. to an external device (not shown) via the output interface 106. The CPU 101 outputs processed image data, etc. to the display device 109 via the output interface 106.

[0018] The bus 107 is a data transmission path for exchanging data between the components of the image forming system 10. All of the components of the image forming system 10 are connected by the bus 107, and for example, the CPU 101 can exchange data with the ROM 103 via the bus 107.

[0019] The external storage device 108 is a storage medium such as an HDD, a memory card, a CF card, an SD card, or a USB memory. The external storage device 108 can also store various data stored in the storage unit 104. The display device 109 is, for example, a CRT display or an LCD (liquid crystal display) display for displaying various processing results by the CPU 101 as images, characters, etc. Note that the display device 109 may be integrated with the image processing device 100, or may include a touch panel that can be operated by a user via touch. Furthermore, the display device 109 may function as a part of the operation unit 110.

[0020] The operation unit 110 is an input device such as a mouse and keyboard, and receives user instructions. The CPU 101 acquires the user instructions input to the operation unit 110 via the input interface 105. The imaging unit 111 captures an input image to be processed by the image processing device 100. The imaging unit 111 includes an image sensor (e.g., CCD, CMOS, etc.) for capturing a recording image formed on a recording medium, for example. The GPU 112 is a processor for performing image processing on the image acquired by the imaging unit 111. The GPU 112 performs calculations on data that it is instructed to process by the CPU 101, and outputs the results to the CPU 101. The CPU 101 writes data to the storage unit 104 via the bus 107, and reads data stored in the storage unit 104.

[0021] The following describes a convolutional neural network (CNN) used in general image processing techniques that apply deep learning technology in this embodiment. CNN is a learning-type image processing technique that convolves an image with a filter generated through learning, and then repeatedly performs nonlinear operations. CNN is also called a model. The filter is a detector for extracting image features, and is also called a local receptive field. An image obtained by the nonlinear operation of convolving the filter with an image is called a feature map. CNN learning is performed based on training data including pairs of input and output images. CNN learning specifically includes generating filter values ​​(parameters) that can be converted with high precision to obtain an output image from an input image, and correcting the parameters.

[0022] When the color information of an image has, for example, RGB color channels, or when the feature map is composed of multiple images, the filter used in the convolution operation has multiple channels corresponding to the number of channels or the number of images. In other words, the convolution filter is expressed as a four-dimensional array including the vertical and horizontal sizes, number of images, and number of channels. The nonlinear operation process after convolving the filter with the image (or feature map) is expressed in units called layers. For example, it is called the nth layer feature map or the nth layer filter. Also, for example, a CNN that repeats the filter convolution and nonlinear operation three times has a three-layer network structure. This process is expressed as shown in Equation 1 below.

[0023]

number

[0024] In Equation 1, Wn is the nth layer filter, bn is the nth layer bias, G is the nonlinear operator, Xn is the nth layer feature map, and * is the convolution operator. Note that the (l) on the right superscript of each parameter indicates that it is the lth filter or feature map. The filter and bias are generated by learning, which will be described later, and are also called network parameters. For example, a sigmoid function and ReLU (Rectified Linear Unit) are used as nonlinear calculation methods. RELU is a function that always sets the output value to 0 when the input value is 0 or less, and sets the output value to the same value as the input value when the input value is greater than 0. ReLU is expressed by Equation 2 below.

[0025]

number

[0026] In Equation 2, X is the input value and G(X) is the output value. Next, CNN learning will be explained. CNN learning involves minimizing the objective function expressed in Equation 3 below using learning data including pairs of learning input images and their corresponding output images.

[0027]

number

[0028] Equation 3 is a loss function for measuring the error between input and output values. In Equation 3, Xi is the i-th input training image data, and Yi is the i-th output training image data. F is a function that integrates the calculations in Equation 1 performed in each layer of the CNN. θ is the network parameter (including the filter and bias). ||F(Xi;θ)-Yi||2 is the L2 norm, which is the square root of the sum of the squares of the vector elements of F and Yi. n is the total number of images in the training data used for training. Because the total number of images in the training data is generally large, stochastic gradient descent uses a randomly selected portion of the training images for CNN training. This reduces the computational load when training a CNN with a large amount of training data. Other well-known methods for minimizing (optimizing) objective functions include the momentum method, AdaGrad method, AdaDelta method, and Adam method. Furthermore, each method has different convergence characteristics, which is known to result in differences in CNN training time. Note that any of the above methods may be used as the optimization method in CNN learning. The Adam method is expressed by the following equation 4.

[0029]

number

[0030] In Equation 4, θit is the ith network parameter at iteration t, and g is the gradient of the loss function L with respect to θit. Furthermore, m and v are moment vectors, α is the basic learning rate, β1 and β2 are hyperparameters, and ε is a small constant. Known examples of CNN-based networks include ResNet in the image recognition field and RED-Net in the super-resolution field. Both networks incorporate multiple layers of convolutional and pooling layers into the CNN, performing multiple filter convolution operations. This improves the accuracy of CNN learning. For example, ResNet has a network structure with a shortcut path through the convolutional layer, enabling a 152-layer network to achieve high-precision recognition approaching that of humans. The reason for the increased accuracy of CNN learning by incorporating multiple layers in the CNN is that the nonlinear relationship between input and output can be accurately represented by repeatedly performing nonlinear operations.

[0031] The following describes the mechanism by which artifacts occur in the prior art with reference to FIGS. 2 and 3. Artifacts refer to data errors and signal distortions that occur during demosaicing. Demosaicing is a process of interpolating color information using pixel values ​​obtained through RGB color filters. To explain the mechanism by which artifacts occur, we will explain the inference process (the process of calculating an output image from an input image) in CNN demosaicing. FIGS. 2(a) and 2(b) are diagrams illustrating how CNN infers an output image based on an input image. In FIG. 2(a), image 201 is an output image inferred by CNN, which has a circle in its center. Images 202 and 203 are input images to be input to CNN. Image 202 has a shape combining a right semicircle and a rectangle, and image 203 has a shape combining a left semicircle and a rectangle. Image 201 is obtained by combining images 202 and 203. Image 204 in FIG. 2(b) is obtained by combining images 205 and 206. A detailed description of FIG. 2(b) is omitted since it is the same as FIG. 2(a).

[0032] First, the image processing device 100 trains a CNN (not shown) for demosaic processing using training data including a large amount of images. At this time, the CNN stores the trained training data in, for example, the storage unit 104. This allows the trained CNN to perform inference to obtain an output image from an untrained input image based on the training data. An example will be described in which the CNN infers an output image based on input images (e.g., image 202 and image 203). Here, the expected output image is image 201 obtained by demosaic processing the input image. To generate image 201, the CNN calls up an image with features similar to image 201 from the training data stored in the storage unit 104.

[0033] If the CNN has learned an image that perfectly matches image 201, the CNN can directly output the learned image. However, if the CNN has not learned an image that perfectly matches image 201, the CNN will call up two or more images similar to image 201 (e.g., image 202 and image 203). The CNN obtains image 201 by combining two or more images using a method such as alpha blending. Alpha blending (hereinafter referred to as blending) is a method of multiplying an image by a coefficient (α value) to give the image a transparent appearance. The CNN can express the color of the circle shown in image 201 by blending the foreground colors of images 202 and 203. The inference process by the CNN to output image 204 in Figure 2(b) is the same as that for image 201, so a description thereof will be omitted.

[0034] FIG. 3 is a diagram illustrating the pixel value range of an image. FIG. 3 shows pixel value range 301, pixel value range 302, and pixel value range 303. A blending method for expressing the color of the circle in image 201 will be described using pixel value range 301. To express pixel value p of the color of the circle in image 201, CNN blends the pixel value (p-ζ) of the foreground color of image 202 and the pixel value (p+η) of the foreground color of image 203 at a ratio (α:β). Here, ratio α is the ratio for blending the foreground color of image 202, and ratio β is the ratio for blending the foreground color of image 203. The blending ratios of the foreground colors of image 202 and image 203, the difference η between pixel value p and the pixel value of image 202, and the difference ζ between pixel value p and the pixel value of image 203 are expressed as α:β=η:ζ. Here, the conditions are that ζ is positive (ζ>0) and η is positive (η>0).

[0035] Returning to the explanation of FIG. 2, the inference process of the CNN when pixel values ​​are saturated, such as the color of the circle in image 204, will be described. Saturation of pixel values ​​refers to a state in which the pixel value is approximately equal to 1, the upper boundary of the pixel value range, i.e., a high brightness state. To output pixel value p of image 204, the CNN calls image 205, which has a dark foreground pixel value (p-ζ), and image 206, which has a foreground pixel value (p+η) approximately equal to 1, as images similar to image 204. Next, pixel value range 302 in FIG. 3 shows the relationship between the pixel values ​​of images 204, 205, and 206. In pixel value range 302, pixel value p of image 204 and pixel value (p+η) of the foreground color of image 206 are 1, so the difference between pixel value p and pixel value (p+η) is 0. In this case, the blending ratio α of the foreground color of image 206 is 0. However, to represent the pixel value p of the color of the circle in image 204, the CNN must blend the pixel value (p + η) of the foreground color of image 206. Here, the proportion α of the foreground color of image 205 must be positive (α > 0). As a result, the CNN cannot determine the value of the proportion β of the foreground color of image 205, and fails to output the pixel value p of image 204. Specifically, the CNN outputs the pixel value p of image 204 as the pixel value when the proportion α is 0, or as the pixel value when the proportion α is positive (α > 0). In other words, depending on the output result of the pixel value p by the CNN, the pixel values ​​of the region representing the color of the circle in image 204 change, resulting in pixel value variations. This region is perceived by the human eye as an artifact.

[0036] The causes of artifacts are explained below. When pixel value p and pixel value (p + η) of image 206 are 1 in pixel value range 302, no image exists with a pixel value greater than pixel value (p + η) (the dashed line portion of pixel value range 302). This means that the pixel value (p + η) of image 206 used to infer pixel value p of image 204 is not in a region greater than the upper boundary of pixel value range 302 used during training. Therefore, this embodiment uses a pixel value range that takes into account the pixel values ​​of the output image so that CNN can accurately infer the pixel values ​​of the output image. This embodiment applies a gain to the input image used for training the CNN to obtain an input image with a pixel value range exceeding 1. This embodiment trains the CNN using an image whose pixel value range has been expanded so that the pixel values ​​of the output image are included in the pixel value range of the image used during training. As a result, in pixel value range 303 of FIG. 3, the pixel value range of the input image used during training can include pixel value p of the output image in its center, thereby reducing artifacts in the output image. Although the method of extending the pixel value range of the input image during learning to suppress the occurrence of artifacts has been described, the pixel value range of the input image input to the CNN before inference may also be reduced. In this embodiment, a case where the pixel value p of the output image obtained by CNN inference is on the upper boundary of the pixel value range during learning has been described as an example. Note that the same method as above may also be applied to a case where the pixel value p of the output image obtained by CNN inference is on the lower boundary of the pixel value range during learning.

[0037] FIG. 4 illustrates the configuration of an image processing device 100. The image processing device 100 includes a teacher image acquisition unit 401, a pixel value range expansion unit 402, a dataset generation unit 403, and a network parameter acquisition unit 404. The image processing device 100 further includes a demosaic learning unit 405, an input image acquisition unit 406, and a demosaic inference unit. In the configuration of the image processing device 100, for example, one functional unit may be divided into multiple functional units, or two or more functional units may be integrated into one functional unit. The configuration of FIG. 4 may also be implemented by two or more image processing devices 100. In this case, the multiple image processing devices 100 are connected via an electrical circuit and a wired or wireless network. The multiple image processing devices 100 can execute each process by cooperating with each other through data communication. While each component shown in FIG. 4 may be described as a processing entity, the function of each functional unit is realized by the CPU 101 executing a computer program corresponding to each functional unit of the image processing device 100. Each functional unit may also be implemented by other hardware.

[0038] The teacher image acquisition unit 401 acquires a teacher image in RGB format from the storage unit 104 or the external storage device 108. The pixel value range expansion unit 402 can expand the pixel value range of the teacher image acquired by the teacher image acquisition unit 401. The data set generation unit 403 performs subsampling on the teacher image whose pixel value range has been expanded, based on an RGB color filter array pattern. Subsampling refers to reducing the amount of image data without significantly degrading image quality by thinning out pixel color information based on a certain rule.

[0039] The network parameter acquisition unit 404 acquires the network parameters of the CNN used during demosaic processing training. The network parameters are the coefficients of each filter that constitutes the CNN. The demosaic learning unit 405 initializes the weight coefficients of the CNN based on the received network parameters, and trains the CNN using pairs of teacher images and student images. The input image acquisition unit 406 acquires the input image 801 for demosaic processing from the storage unit 104. The demosaic inference unit 407 uses the CNN that has already trained the training data by the demosaic learning unit 405.

[0040] FIG. 5 is a diagram illustrating a series of steps in image processing using the functions of the image processing device 100. Below, an overview of the processing using the functions of the image processing device 100 will be described with reference to FIG. 5. In S501, the teacher image acquisition unit 401 acquires a teacher image 803 in RGB format. The teacher image is generated according to the method described in Non-Patent Document 1. FIG. 8 illustrates how the teacher image acquisition unit 401 acquires an image. As shown in FIG. 8, the imaging unit 111 acquires an input image 801 containing RGB and performs simple demosaic processing on it to generate an RGB image 802. The imaging unit 111 reduces the image of the RGB image 802 to generate a teacher image 803. The simple demosaic processing is, for example, bilinear interpolation, but other demosaic processing methods may also be used. Furthermore, the color filter array in this embodiment is a Bayer array, but may also be a color filter array such as X-Trans.

[0041] Furthermore, the teacher image 803 in RGB format may be acquired by a method other than the method of Non-Patent Document 1. For example, the teacher image acquisition unit 401 may read out the teacher image 803 captured in advance by the imaging unit 111 from the storage unit 104. The teacher image acquisition unit 401 may also acquire the teacher image 803 in RGB format by capturing images while shifting the position of the image sensor of the imaging unit 111. Next, the teacher image acquisition unit 401 normalizes the pixel values ​​of the acquired teacher image 803 so that they fall within a pixel value range having an interval from 0 to 1, using the method shown in the following equation 5.

[0042]

number

[0043] Here, Xinput is the pixel value of the acquired teacher image 803, nmax is the maximum pixel value of Xinput, and Xnorm is the pixel value of the normalized teacher image 803. For example, if Xinput is an 8-bit image, nmax is 255, and if Xinput is a 14-bit image, nmax is 16383. The teacher image acquisition unit 401 transmits the normalized teacher image 803 to the pixel value range extension unit 402. In S502, the pixel value range extension unit 402 extends the pixel value range of the received teacher image 803 according to the following equation 6.

[0044]

number

[0045] Here, Xrange is the pixel value range of the teacher image 803 after the pixel value range has been extended, ρinf is the downward extension width of the pixel value range, and ρsup is the upward extension width of the pixel value range. The conversion formula for obtaining the extended pixel value range is expressed as f(Xnorm). FIG. 9 is a diagram illustrating the relationship between extension and contraction of the pixel value range of the teacher image 803. FIG. 9 shows pixel value ranges 901, 902, 903, 904, and 905. In FIG. 9, the pixel value range 901 of the teacher image 803 has pixel values ​​ranging from a minimum value of 0 to a maximum value of 1. On the other hand, the pixel value range 902 obtained by extending the pixel value range 901 has pixel values ​​ranging from a minimum value of -ρinf to a maximum value of 1+ρsup.

[0046] The downward extension width ρ inf and the upward extension width ρ sup may be determined based on a predetermined value. Furthermore, the extension width of the pixel value range may be different for each teacher image 803, and these extension widths may be determined based on random numbers. The pixel value range extension unit 402 may determine the extension width of the pixel value range based on the characteristics of the teacher image 803. For example, when extending the pixel value range of the teacher image 803 by a predetermined extension width, the pixel value range extension unit 402 calculates the proportion of pixels in the teacher image 803 whose pixel values ​​deviate from the range from 0 to 1. The pixel value range extension unit 402 may determine the extension width based on, for example, whether the proportion of pixels deviating from the pixel value range exceeds a predetermined threshold.

[0047] In FIG. 9 , pixel value range 904 has a zero point Z (black point) within the pixel value range from a minimum value of 0 to a maximum value of 1. A pixel value range 905 obtained by extending pixel value range 904 has a zero point f(z) (black point) within the pixel value range from a minimum value of −ρinf to a maximum value of 1+ρsup. When extending the pixel value range, the pixel value range extension unit 402 extends the pixel value range 904 while matching the zero point z of pixel value range 904 with the zero point f(z) of pixel value range 905. The zero point is the pixel value when the amount of light received by the image sensor of the image capture unit 111 is 0. To prevent pixel values ​​from becoming negative due to additive noise, the zero point is set to a value such as z=1 / 32. The pixel value range extension unit 402 determines the extension width so that the value of the zero point f(z) after the pixel value range extension is the same as the zero point z before the extension. The pixel value range extension unit 402 determines the upward extension width ρsup in accordance with Equation 7.

[0048]

number

[0049] 9, the zero points of a pixel value range 904 before the pixel value range is extended and a pixel value range 905 after the pixel value range is extended are the same. As a result, this embodiment can improve the inference accuracy for pixel values ​​that exist on the boundary (minimum value 0 or maximum value 1) of the pixel value range of the teacher image 803 during inference by CNN. The pixel value range extension unit 402 transmits the teacher image 803 with its pixel value range extended to the dataset generation unit 403.

[0050] In S503, the dataset generation unit 403 generates an input image 801 by subsampling the received teacher image 803 based on a color filter array pattern. FIG. 10 shows the flow of generating an input image based on the color filter array of the image sensor of the imaging unit 111. In FIG. 10, the dataset generation unit 403 obtains a student image 1004 by subsampling an R component 1001, a G component 1002, and a B component 1003 of the teacher image 803 based on a color filter array 1005. The dataset generation unit 403 transmits an image set including the generated student image 1004 and teacher image 803 to the demosaic learning unit 405.

[0051] Returning to the explanation of FIG. 5, in S504, the network parameter acquisition unit 404 acquires network parameters for training the CNN through demosaic processing. The network parameters are set as random numbers following a He normal distribution. The He normal distribution is a normal distribution with a mean of 0 and a variance of σh, as shown in the following equation 8.

[0052]

number

[0053] Here, mN is the number of inter-layer neurons constituting the CNN. Note that the network parameters may be determined by a method other than Equation 8. The network parameter acquisition unit 404 transmits the acquired network parameters to the demosaic learning unit 405. In S505, the demosaic learning unit 405 initializes the CNN weight coefficients with the received network parameters and then trains the CNN using the received image set. The demosaic learning unit 405 trains the CNN using a teacher image 803. FIG. 11 shows the structure of the CNN and a learning sequence of this embodiment. In FIG. 11, the CNN includes a student image 1004, missing images 1101a to 1101c, a filter 1102, a connection layer 1103, and output images 1104a to 1104c. The CNN includes multiple filters 1102 to perform calculations based on Equation 1. When a student image 1004 is input to the CNN, the CNN converts the student image 1004 into a three-channel (RGB) missing image 1101a, a missing image 1101b, and a missing image 1101c. The R channel missing image 1101a contains only the R component pixels (colored portion) of the student image 1004, and the pixel values ​​of the other pixel components (GB) are set to 0 (white portion).

[0054] The missing image 1101b of the G channel includes only the pixels of the G component (colored portion) of the student image 1004, and the pixel values ​​of the other pixel components (RB) are set to 0 (white portion). The missing image 1101c of the B channel includes only the pixels of the B component (colored portion) of the student image 1004, and the pixel values ​​of the other pixel components (RG) are set to 0 (white portion). Note that, in the missing images 1101a to 1101c, pixel values ​​of the missing portions (portions with pixel values ​​of 0) in each missing image may be interpolated using a method such as bilinear interpolation. Next, the CNN calculates a feature map by sequentially applying the filter 1102 to the missing images 1101a to 1101c.

[0055] In FIG. 11, the concatenation layer 1103 concatenates the calculated feature map with missing images 1101a to 1101c in the channel direction. If the number of channels in the feature map and missing images is n1 and n2, respectively, the number of channels in the concatenation result is n1+n2. The CNN outputs the concatenation result through filter 1102 to three channels, obtaining output images 1104a, 1104b, and 1104c. The CNN calculates the residual between each output image and the training image 803 and averages the residuals across the entire image to calculate the value of the loss function. The CNN updates the network parameters based on the calculated loss function value, for example, using backpropagation.

[0056] After updating the network parameters, in S506 of FIG. 5, the CPU 101 determines whether CNN learning is complete. The CPU 101 determines whether the number of iterations of learning (updating the network parameters) exceeds a threshold, for example, as a criterion for determining whether CNN learning is complete. Note that the criterion for determining whether CNN learning is complete is not limited to the above. For example, the criterion may be whether the residual (the difference between the teacher image 803 and each output image) before and after updating the network parameters is smaller than a specified value. Alternatively, the criterion may be whether each residual between each output image and the teacher image 803 is smaller than a specified value. If CNN learning is not complete, the process returns to S501, the dataset generation unit 403 generates the next image set, and the demosaic learning unit 405 resumes learning based on the image set (No in S506). If CNN learning is complete, the demosaic learning unit 405 transmits the updated network parameters to the demosaic inference unit 407 (Yes in S506).

[0057] In S507, the input image acquisition unit 406 captures an input image 801 for demosaic processing using the imaging unit 111. Note that the input image 801 may be an image captured in advance by the imaging unit 111 and stored in the storage unit 104. The input image acquisition unit 406 transmits the acquired input image 801 to the demosaic inference unit 407. In S508, the demosaic inference unit 407 uses the CNN used by the demosaic learning unit 405. The demosaic inference unit 407 initializes the network parameters of the CNN with the network parameters received from the demosaic learning unit 405. The demosaic inference unit 407 performs inference on the input image 801 using the CNN with the updated network parameters, in the same manner as the demosaic learning unit 405 performed during learning. As a result, the demosaic inference unit 407 obtains output images 1104a to 1104c as demosaic images.

[0058] In this embodiment, a method has been described in which the pixel value range during learning is extended and then the CNN is trained so that the pixel values ​​of the input image 801 at the time of inference are included in the pixel value range during learning, in order to accurately infer the pixel values ​​of the input image 801. Note that the pixel value range during learning may be a pixel value range in which either the upper or lower boundary of the pixel value range is extended within a range that can include the pixel values ​​at the time of inference.

[0059] As described above, according to the first embodiment, a first image is acquired to be used for training a model that trains an image based on the luminance of the image. If the first luminance of the first image exceeds a threshold, a second image can be generated as an image having a luminance different from the first luminance. According to the first embodiment, parameters for training the image held by the model can be corrected based on the first luminance and the second luminance of the second image. This can suppress color degradation that occurs during image demosaicing and improve the color quality of the image.

[0060] (Second embodiment) The second embodiment will be described below in terms of differences from the first embodiment.

[0061] The first embodiment described an example in which a CNN is trained on a teacher image 803 whose pixel value range is expanded relative to that of an input image 801 at the time of inference. In the second embodiment, a CNN is trained using a teacher image 803 whose pixel value range is not expanded, and then trained on a teacher image 803 whose pixel value range is expanded. The second embodiment includes pre-training in which the pixel value range is not expanded and main training in which the pixel value range is expanded. This enables the second embodiment to train a CNN that can infer pixel values ​​that may exist at the upper and lower boundaries and in the center of the pixel value range. The second embodiment can improve the robustness of the inference of pixel values ​​of the input image 801.

[0062] FIG. 12 is a diagram illustrating the flow of learning in this embodiment. The learning flow will be described below with reference to FIG. 12. In S1201 of FIG. 12, as pre-learning for the CNN, the pixel value range extension unit 402 does not extend the lower extension width ρinf and the upper extension width ρsup of the teacher image 803. Note that the pixel value range extension unit 402 may extend the pixel value range of the teacher image 803 by an extension width smaller than that in the first embodiment. From here on, in this embodiment, the CNN is trained using the teacher image 803 in the same manner as in the first embodiment. As a result, in this embodiment, the CNN can learn pixel values ​​corresponding to the center of the pixel value range. After CNN learning is complete, the demosaic learning unit 405 outputs the network parameters obtained by pre-learning.

[0063] Next, as main training for the CNN, the pixel value range expansion unit 402 generates a teacher image 803 in which the pixel value range is expanded to the same extent as in the first embodiment, and the demosaic learning unit 405 uses this teacher image 803 to train the CNN. At this time, the network parameter acquisition unit 404 acquires the network parameters output in the pre-training. The demosaic learning unit 405 trains the CNN using these network parameters as initial values. This allows the demosaic learning unit 405 to improve the estimation accuracy of pixel values ​​of the input image 801 at the upper or lower boundary of the learned pixel value range. Note that the demosaic learning unit 405 may acquire the trained network parameters from the storage unit 104, a storage medium (not shown), or the web (not shown). Acquiring the trained network parameters may be pre-training. In other words, in main training, the network parameter acquisition unit 404 may use trained network parameters acquired without performing pre-training.

[0064] As described above, according to the second embodiment, two-stage CNN learning can learn pixel values ​​of an image including the entire region of the upper or lower boundary and the center of the pixel value range. This can further reduce artifacts that occur during inference. Furthermore, the pre-learning is learning using a teacher image 803 that does not extend the pixel value range, which is easy for CNN to learn. The main learning is learning using a teacher image 803 that extends the pixel value range, which is difficult for CNN to learn. According to the second embodiment, a high artifact suppression effect can be achieved with low computational cost based on curriculum learning, such as two-stage learning. The number of learning stages is not limited to the two stages described above, and may be a number corresponding to multiple extension widths of the pixel value range. Furthermore, multiple stages that dynamically increase the extension width of the pixel value range may be set during one-stage learning.

[0065] As described above, according to the second embodiment, the first image is an image that the model has already learned, and when the third luminance exceeds the first luminance, the generating means generates a fourth image having a luminance different from the third luminance. According to the second embodiment, the correcting means can correct parameters used by the model to learn the image based on the first luminance and the fourth luminance of the fourth image. This can suppress color degradation that occurs during image demosaicing and improve the color quality of the image.

[0066] (Third embodiment) Hereinafter, the third embodiment will be described with respect to differences between the first and second embodiments.

[0067] In the first and second embodiments, the pixel value range of a teacher image 803 during CNN training is extended relative to the pixel value range of an input image 801 during inference. In the third embodiment, an example is described in which the pixel value range during inference is reduced so that the pixel value range during training is included in the pixel value range during inference without extending the pixel value range during training. FIG. 6 is a block diagram showing the configuration of an image processing device 100 according to the third embodiment. In FIG. 6, the image processing device 100 includes a pixel value range reduction unit 601 and a pixel value range restoration unit 602. The pixel value range reduction unit 601 reduces the pixel value range of an input image 801 acquired by the input image acquisition unit 406. The pixel value range restoration unit 602 restores each output image inferred by the demosaic inference unit 407 to the same pixel value range before reduction by the pixel value range reduction unit 601. FIG. 7 is a diagram illustrating a series of image processing flows using the functions of the image processing device 100. The configuration of the image processing device 100 according to this embodiment will be described below with reference to FIGS. 6 and 7.

[0068] This embodiment differs from the first and second embodiments in that the pixel value range of the teacher image 803 is not expanded at all during learning. The pixel value range during learning in this embodiment has a section from a minimum value of 0 to a maximum value of 1, as shown in pixel value range 901 in FIG. 9. In S701 of FIG. 7, the pixel value range reduction unit 601 reduces the pixel value range of the input image received from the input image acquisition unit 406 based on Equation 6. Here, the lower reduction width ρinf and the upper reduction width ρsup may be predetermined values ​​that are defined in advance. The conditions at this time are ρinf<0 and ρsup<0. The pixel value range reduction unit 601 performs reduction processing on the input image to obtain the following results.

[0069] As shown in FIG. 9 , a pixel value range 901 ranging from a minimum value of 0 to a maximum value of 1 is reduced to a pixel value range 903 ranging from a minimum value of −ρinf to a maximum value of 1+ρsup. The pixel value range reduction unit 601 transmits an image having the pixel value range 903 to the demosaic inference unit 407. In S508, the demosaic inference unit 407 inputs the image having the pixel value range 903 to a CNN to obtain an output image (demosaic image). The demosaic inference unit 407 transmits the output image to the pixel value range restoration unit 602. In S702, the pixel value range restoration unit 602 restores the pixel value range of the output image received from the demosaic inference unit 407 based on the following equation (9).

[0070]

number

[0071] Here, Xdemosaicked is the pixel value range of the output image (demosaic image). Xrecover is the pixel value range of the output image restored to the state before the pixel value range was reduced. Equation 9 is the inverse transform of Equation 6. Equation 9 restores the pixel value range of the demosaic image so that it approximately matches the pixel value range of the input image acquired by the input image acquisition unit 406. In this embodiment, the lower reduction width ρinf and the upper reduction width ρsup may be determined by selecting from a plurality of reduction widths obtained by variously changing the reduction rate of the pixel value range. A specific method for determining the lower reduction width ρinf and the upper reduction width ρsup is described below.

[0072] For example, the teacher image acquisition unit 401 acquires an RGB teacher image 803 and an input image 801 generated by subsampling the teacher image 803. Next, the demosaic inference unit 407 performs demosaic processing on the input image 801, for which the lower reduction width ρinf and the upper reduction width ρsup are set to small values. The demosaic inference unit 407 determines the reduction rate for each reduction width based on the occurrence of artifacts in the output image obtained by the demosaic processing. The occurrence of artifacts is evaluated by the average error between the pixel values ​​of the output image and the pixel values ​​of the teacher image 803 at the upper or lower boundary of the pixel value range. If the demosaic inference unit 407 determines that the occurrence (frequency) of artifacts exceeds a specified value, the pixel value range reduction unit 601 reduces the pixel value range by increasing each of the lower reduction width ρinf and the upper reduction width ρsup by 10%.

[0073] The demosaic inference unit 407 again performs demosaic processing on the input image 801 having the reduced pixel value range. These reduction and demosaic processes are repeated multiple times. If the demosaic inference unit 407 determines that the artifact occurrence rate does not exceed a specified value, the pixel value range reduction unit 601 terminates the reduction of the pixel value range. The acquired image at this time becomes the final output image. Note that the increase rate of the reduction rate of the lower reduction width ρinf and the upper reduction width ρsup does not have to be 10%. The method for evaluating the artifact occurrence rate is not limited to the above and may be, for example, a method in which the user directly visually checks the range. This embodiment has described an example in which the pixel value range of the teacher image 803 is not extended during learning, but the pixel value range of the input image 801 during inference is reduced. Note that the pixel value range of the teacher image 803 may be extended during learning, and the pixel value range of the input image 801 during inference may be further reduced.

[0074] As described above, according to the third embodiment, a fifth image that has already been trained by a model that trains images based on the luminance of the image and a sixth image that the model has not trained can be acquired, and it can be determined whether the sixth luminance of the sixth image exceeds the fifth luminance of the fifth image. According to the third embodiment, the sixth image can be generated as a seventh image according to the determination result, as an image having a luminance different from the fifth luminance. According to the third embodiment, the seventh luminance of the seventh image can be estimated using a model that has the training results of the fifth image. As a result, according to the third embodiment, color degradation that occurs during image demosaicing can be suppressed, and the color quality of the image can be improved.

[0075] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0076] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0077] 10: Image forming system, 100: CPU, 101: RAM, 102: ROM, 103: Operation unit, 104: Display unit, 105: Storage unit

Claims

1. A storage means for storing a model for performing demosaic processing on a mosaic image of a predetermined color filter array; an acquisition means for acquiring a teacher image; an extension means for extending the pixel value range of the teacher image so that pixel values ​​of an output image obtained by inputting a mosaic image into the model are included in the pixel value range of the teacher image; a generating means for generating a student image, which is a mosaic image corresponding to the teacher image and has a pixel value range that substantially matches the pixel value range expanded by the expanding means; a correction means for correcting parameters of the model based on a residual between an output image obtained by inputting the student image into the model and the teacher image whose pixel value range has been expanded; An image processing device comprising:

2. the extension means determines the width by which the pixel value range is extended based on whether or not a ratio of pixels that deviate from the pixel value range exceeds a threshold value. The image processing device according to claim 1 .

3. The method further comprises inferring an output image obtained by demosaicing the mosaic image using the model having the parameters corrected by the correction means.

3. The image processing device according to claim 1 or 2.

4. The method further comprises: storing parameters of the model, including the parameters corrected by the correction means. The image processing device according to claim 1 .

5. the decompression means decompresses at least one of the pixel values ​​on the upper limit side and the pixel values ​​on the lower limit side of the pixel value range. The image processing device according to claim 1 .

6. An imaging device; A plurality of image processing devices according to any one of claims 1 to 5; An image processing system comprising:

7. An image processing method executed by an image processing device, a storing step of storing a model for performing demosaicing on a mosaic image of a predetermined color filter array; an acquisition step of acquiring a teacher image; an extension step of extending the pixel value range of the teacher image so that pixel values ​​of an output image obtained by inputting a mosaic image into the model are included in the pixel value range of the teacher image; a generating step of generating a student image, which is a mosaic image corresponding to the teacher image and has a pixel value range that substantially matches the pixel value range expanded by the expanding step; a correction step of correcting parameters of the model based on a residual between an output image obtained by inputting the student image into the model and the teacher image whose pixel value range has been expanded; An image processing method comprising:

8. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 5.

Images