Image processing apparatus and image processing method

Abstract

An image processing apparatus according to an embodiment includes a remosaic processing unit and an image processing unit. The remosaic processing unit converts an array of three primary color pixels including red, green, and blue pixels as three primary colors of light and pixels of colors other than the three primary colors in an input image into a Bayer array. And generating a first three primary color image including color information of the three primary colors and not including color information of colors other than the three primary colors. The image processing unit generates a second three-primary color image from the input image, the second three-primary color image including color information for the three primary colors and color information for colors other than the three primary colors.

Description

Technical Field

[0001] The present disclosure relates to an image processing device and an image processing method. Background Art

[0002] A digital camera includes an image sensor. The image sensor includes multiple imaging elements corresponding to pixels that capture images. Each imaging element is provided with, for example, color filters that selectively transmit light beams of specific wavelengths, such as red, green, and blue, which are the three primary colors of light.

[0003] The image sensor photoelectrically converts the light beams transmitted through the color filters through its various imaging elements to generate a three-primary-color image. This three-primary-color image is subsequently output to the camera system. The camera system performs color information supplementation and correction processing on the three-primary-color image input from the image sensor, and then outputs the processed three-primary-color image to a display device, storage device, or the like.

[0004] As an array of color filters, the current mainstream is the Bayer array, in which green filters and blue filters are alternately arranged in the row direction of pixels, and green filters and red filters are alternately arranged in the column direction of pixels.

[0005] In recent years, for the purpose of realizing multifunctional and high-performance digital cameras, image sensors in which color filters are arranged to form an array different from the Bayer array have been developed. Examples of arrays different from the Bayer array include Quad Bayer Coding (QBC).

[0006] In QBC, pixel blocks of four adjacent pixels provided with green filters are arranged alternately with pixel blocks of four adjacent pixels provided with blue filters in the row direction. Also, in QBC, pixel blocks of four adjacent pixels provided with green filters are arranged alternately with pixel blocks of four adjacent pixels provided with red filters in the row direction.

[0007] As a result, image sensors can suppress degradation in image resolution, for example, at night or in low-light conditions. However, subsequent camera systems typically assume a Bayer array as input image and are therefore unable to process images using arrays other than the Bayer array. Consequently, developing camera systems capable of processing images using arrays other than the Bayer array requires significant time and cost.

[0008] Image sensors can further improve image resolution by converting the pixels of images captured by QBC to a Bayer array during daytime or under high illumination. The process of converting an array other than the Bayer array to the Bayer array is called re-mosaicing (see, for example, Patent Document 1).

[0009] Furthermore, when the colors that an image sensor can receive are the three primary colors of red, green, and blue, faithful reproduction of natural colors is difficult because the spectral characteristics differ from those of the naked eye. Therefore, if the image sensor is provided with a color filter that can receive light beams of colors other than the three primary colors, the reproducibility of natural colors can be improved.

[0010] Reference List

[0011] Patent Literature

[0012] Patent Document 1: JP 2013-66146A Summary of the Invention

[0013] Technical issues

[0014] However, conventional camera systems cannot process images that include pixels for the three primary colors and pixels for colors other than the three primary colors, which are not part of the Bayer array. Therefore, image sensors must perform a re-mosaicing process on images that include pixels for the three primary colors and pixels for colors other than the three primary colors, and then output the resulting image to the camera system. However, when re-mosaicing is performed on images that include pixels for the three primary colors and pixels for colors other than the three primary colors, information about colors other than the three primary colors is lost, and color reproducibility degrades.

[0015] Therefore, the present disclosure proposes an image processing apparatus and an image processing method capable of improving color reproducibility while performing re-mosaic processing.

[0016] Solution to the problem

[0017] An image processing apparatus according to an embodiment includes a re-mosaic processing unit and an image processing unit. The re-mosaic processing unit converts an array of pixels including three primary colors (red, green, and blue) and pixels of colors other than the three primary colors in an input image into a Bayer array, and generates a first three-primary color image including color information of the three primary colors and excluding color information other than the three primary colors. The image processing unit generates a second three-primary color image including color information of the three primary colors and color information other than the three primary colors from the input image. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic configuration diagram of an image processing apparatus according to an embodiment.

[0019] Figure 2 is a functional block diagram showing a configuration example of an image processing apparatus according to an embodiment.

[0020] Figure 3 is an illustrative view of a simplified demosaicing process according to an embodiment.

[0021] Figure 4 is a diagram showing an example of a 2D-LUT according to an embodiment.

[0022] Figure 5 This is a schematic configuration diagram of an image processing device according to Modification 1 of the embodiment.

[0023] Figure 6 is a functional block diagram showing a configuration example of an image processing apparatus according to a first modification example of the embodiment.

[0024] Figure 7 : is a functional block diagram showing a configuration example of an image processing apparatus according to a second modification example of the present embodiment.

[0025] Figure 8 It is an explanatory diagram of a modified example of an input image according to the embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in each of the following embodiments, the same reference numerals are given to the same parts and repeated descriptions are omitted.

[0027] [1. Outline Configuration of Image Processing Device According to Embodiment]

[0028] First, refer to Figure 1 An outline configuration of an image processing apparatus according to an embodiment is described. Figure 1 1 is a schematic configuration diagram of an image processing apparatus 1 according to an embodiment. Figure 1 As shown, the image processing apparatus 1 is connected between an imaging unit 10 and a camera system 20. The imaging unit 10 has a plurality of imaging elements corresponding to respective pixels of a captured image.

[0029] A plurality of imaging elements are arranged in a matrix and provided with color filters each selectively transmitting light of a specific wavelength. The imaging element of a general imaging unit is provided with a color filter that selectively transmits light of any one of the three primary colors of light (red, green, and blue).

[0030] However, in the subsequent camera system 20, it is difficult to faithfully reproduce natural colors using only the three colors of red, green, and blue that the imaging unit 10 can receive. Therefore, the imaging element included in the imaging unit 10 of this embodiment includes a color filter that selectively transmits light of any one of the three primary colors of cyan, magenta, and yellow.

[0031] That is, each imaging element is provided with a color filter that selectively transmits light of any one color of red, green, blue, cyan, magenta, and yellow. Therefore, the array of color filters in the imaging unit 10 is different from the Bayer array.

[0032] In the following description, red is described as R, green is described as G, blue is described as B, cyan is described as C, magenta is described as M, and yellow is described as Y.

[0033] Each imaging element photoelectrically converts light incident through the lens and each color filter into a signal charge corresponding to the amount of light received, generates a captured image including color information of six colors of R, G, B, C, Y, and M (hereinafter described as "input image P1"), and inputs the generated image to the image processing device 1.

[0034] In this embodiment, the arrangement of the color filters included in the imaging unit 10 is different from the above-mentioned Bayer arrangement. Therefore, the color array of the pixels in the input image P1 is also different from the Bayer array.

[0035] On the other hand, assuming that the array of colors of pixels in an image to be processed is the Bayer array, the camera system 20 connected at a subsequent stage cannot process an input image P1 having an array other than the Bayer array.

[0036] Therefore, the image processing apparatus 1 includes a regeneration processing unit 2. The re-mosaic processing unit 2 converts an array of R, G, and B pixels in an input image P1 including pixels of R, G, B, C, M, and Y into a Bayer array, generates a first three-primary color image P2 including color information of R, G, and B and excluding color information of C, M, and Y, and outputs the generated image to the camera system 20.

[0037] Thus, the camera system 20 can process the first three-primary color image P2 input from the image processing device 1. The camera system 20 includes a demosaic processing unit 21, a white balance (hereinafter referred to as "WB") processing unit 22, a color conversion matrix (hereinafter referred to as "CCM") processing unit 23, and a gamma correction processing unit 24.

[0038] The demosaic processing unit 21 complements the color information of each pixel arranged in the Bayer array in the first three-primary-color image P2 based on the color information of adjacent pixels, and outputs the resulting image to the WB processing unit 22. The WB processing unit 22 performs processing to adjust the reference of white color of the first three-primary-color image P2 so that the white color of the first three-primary-color image P2, which is complementary to the color information input from the demosaic processing unit 21, becomes white regardless of the color temperature of the light source, and outputs the resulting image to the CCM processing unit 23. A specific embodiment of the WB processing will be described later.

[0039] The CCM processing unit 23 convolves a preset color conversion matrix with the color information of each pixel in the first three-primary-color image P2 that has undergone WB processing, converts the hue of the first three-primary-color image P2 to the hue desired by the user, and outputs the resulting image to the gamma correction processing unit 24. A specific example of CCM processing will be described later.

[0040] The gamma correction processing unit 24 corrects the color information of the first three-primary-color image P2 to color information corresponding to the gamma values ​​inherent to the display device, so that the display device can correctly display the colors of the first three-primary-color image P2 after CCM processing. The gamma correction processing unit 24 outputs the gamma-corrected first three-primary-color image P2 to, for example, a display device or a storage device.

[0041] As described above, the camera system 20 can process the first three primary color image P2 , but since the first three primary color image includes only color information of R, G, and B and does not include color information of C, M, and Y, there is room for improving reproducibility of natural colors.

[0042] Therefore, the image processing device 1 of this embodiment includes, in addition to the re-mosaic processing unit 2, an image processing unit 3, which generates, based on the input image P1, a second three-primary color image including color information other than R, G, and B. For example, the image processing unit 3 generates, from the input image P1, a second three-primary color image including color information of R, G, B, C, M, and Y.

[0043] Then, for example, the image processing unit 3 generates information required to reflect the color information of C, M, and Y from the second three primary color image in the camera system 20 to the first three primary color image P2 at a subsequent stage, and outputs the generated information to the camera system 20 .

[0044] For example, the image processing unit 3 calculates the count of the color conversion matrix to be multiplied by the color information of R, G, and B (hereinafter, described as “parameter PM”) to reflect the color information of C, M, and Y in the CCM processing unit 23 of the camera system 20, and outputs the count to the CCM processing unit 23.

[0045] As a result, the image processing device 1 can reflect the color information of C, M, and Y, for example, on the first three-primary color image P2 in the subsequent camera system 20. Therefore, even in the camera system 20 that can only process images in the Bayer array, the image processing device 1 can improve the reproducibility of natural colors in the input image P1 while performing the re-mosaicing process.

[0046] [2. Configuration Example of Image Processing Device]

[0047] Next, we will refer to Figure 2A configuration example of the image processing apparatus 1 according to the embodiment is described. Figure 2 1 is a functional block diagram showing a configuration example of the image processing apparatus 1 according to the embodiment.

[0048] like Figure 2 As shown, the image processing apparatus 1 includes a re-mosaic processing unit 2 and an image processing unit 3. The image processing unit 3 includes a six-color WB processing unit 31, a reduction demosaic processing unit 32, a 3×3 CCM processing unit 33, a 2D-LUT processing unit 34, a 3×6 CCM processing unit 35, a 2D-LUT processing unit 36, and a parameter estimation processing unit 37.

[0049] Some or all of the re-mosaic processing unit 2, the six-color WB processing unit 31, the reduction and demosaicing processing unit 32, the 3×3 CCM processing unit 33, the 2D-LUT processing unit 34, the 3×6 CCM processing unit 35, the 2D-LUT processing unit 36 ​​and the parameter estimation processing unit 37 are configured by hardware such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0050] Note that the re-mosaic processing unit 2, six-color WB processing unit 31, reduction demosaic processing unit 32, 3×3 CCM processing unit 33, 2D-LUT processing unit 34, 3×6 CCM processing unit 35, 2D-LUT processing unit 36 ​​and parameter estimation processing unit 37 can be configured by software.

[0051] In this case, the image processing device 1 includes a microcomputer including a central processing unit (CPU), a read-only memory (ROM), a random access memory (RAM), etc., and various circuits. The CPU executes a program stored in the ROM using the RAM as a work area to function as a re-mosaic processing unit 2, a six-color WB processing unit 31, a reduction demosaic processing unit 32, a 3×3 CCM processing unit 33, a 2D-LUT processing unit 34, a 3×6 CCM processing unit 35, a 2D-LUT processing unit 36, and a parameter estimation processing unit 37.

[0052] The re-mosaic processing unit 2 converts the pixel array of the input image P1 having an array different from the Bayer array into a Bayer array, generates a first three-primary color image P2 including color information of R, G, and B and excluding color information of C, M, and Y, and outputs a second three-primary color image P3 to the camera system 20.

[0053] The six-color WB processing unit 31 adjusts the reference of white in the first three-primary-color image P2 so that the white in the input image P1 becomes white regardless of the color temperature of the light source. Specifically, the six-color WB processing unit 31 multiplies the gains (wbr) using the following formula (1) so that the pixel values ​​of the color information of R, G, B, C, M, and Y become the same pixel value in the white area of ​​the input image P1.

[0054]

[0055] The input image P1 after the six-color WB processing is output to the reduction demosaic processing unit 32. Note that the WB processing unit 22 of the camera system 20 performs WB processing on the first three primary color image P2 using the expressions of R, G, and B in formula (1).

[0056] like Figure 3 As shown, the reduction demosaic processing unit 32 specifies a block of a predetermined size indicated by a thick line frame in the input image P1 after the six-color WB processing. Then, the reduction demosaic processing unit 32 performs a multi-color multi-image processing on each color in each block (in Figure 3 The pixel values ​​of R) in the embodiment shown in are averaged to aggregate into one pixel.

[0057] For G, B, C, M, and Y, the reduction demosaic processing unit 32 similarly averages the pixel values ​​in each block to aggregate into one pixel, thereby generating a reduced image including color information of six planes (six colors), and outputs the reduced image to the 3×6 CCM processing unit 35.

[0058] Furthermore, the reduction demosaic processing unit 32 averages the pixel values ​​in each block for R, G, and B to aggregate into one pixel, thereby generating a reduced image including color information of three planes (three colors), and outputs the reduced image to the 3×3 CCM processing unit 33 .

[0059] The reduction demosaic processing unit 32 can generate a reduced image having a smaller number of pixels relative to the input image P1 by increasing the size of a designated block. As a result, the image processing unit 3 can reduce the amount of memory used in image processing in subsequent stages.

[0060] The 3×3 CCM processing unit 33 performs 3×3 CCM processing by convolving the 3×3 matrix of formula (2) with the R, G, and B values ​​of the pixels in the reduced image (which includes the color information of R, G, and B but does not include the color information of C, M, and Y) to convert the R, G, and B values ​​into values ​​close to the true values ​​of R, G, and B.

[0061]

[0062] The 3×3 CCM processing unit 33 outputs the reduced image after the 3×3 CCM processing to the 2D-LUT processing unit 34. The CCM processing unit 23 of the camera system 20 performs 3×3 CCM processing similar to that of the 3×3 CCM processing unit 33.

[0063] The 3×6 CCM processing unit 35 performs 3×6 CCM processing of convolving the 3×6 matrix of formula (3) with the R, G, B, C, M, and Y values ​​of the pixels in the reduced image including the color information of R, G, B, C, M, and Y to convert the R, G, B, C, M, and Y values ​​into values ​​close to the true values ​​of R, G, and B.

[0064]

[0065] The 3×6 CCM processing unit 35 outputs the reduced image after the 3×6 CCM processing to the 2D-LUT processing unit 36 ​​.

[0066] The 2D-LUT processing unit 34 and the 2D-LUT processing unit 36 ​​convert the pixel values ​​of R, G, and B of the reduced image input thereto into values ​​of the HLS space including three components of hue (H), brightness (L), and saturation (S).

[0067] Then, the 2D-LUT processing unit 34 and the 2D-LUT processing unit 36 ​​use, for example, Figure 4 The lookup table (LUT) shown in finely adjusts the values ​​of hue (H) and saturation (S) to reproduce colors that cannot be expressed by CCM alone. Figure 4 In the case of the LUT shown in , for example, the values ​​of hue (H) with saturation (S) of 11 to 13 are converted into 65, 70, and 65, respectively.

[0068] The 2D-LUT processing unit 34 outputs the three-primary color image including color information of R, G and B but not including color information of C, M and Y to the parameter estimation processing unit 37. The 2D-LUT processing unit 36 ​​outputs the three-primary color image including color information of R, G, B, C, M and Y to the parameter estimation processing unit 37.

[0069] In this manner, the image processing unit 3 divides the input image P1 into a plurality of blocks, averages the color information of a plurality of pixels of the same color for each of the colors R, G, B, C, M, and Y included in the blocks, and aggregates them into one pixel, thereby generating a reduced image of the input image P1. The image processing unit 3 then generates a second three-primary color image P3 including color information of R, G, B, C, M, and Y from the reduced image.

[0070] In this manner, the image processing unit 3 divides the input image P1 into a plurality of blocks, averages the color information of a plurality of pixels of the same color for each of the colors R, G, and B included in the blocks, and aggregates them into one pixel, thereby generating a reduced image of the input image P1. The image processing unit 3 then generates a third three-primary color image P4 that includes color information of R, G, and B and does not include color information of C, M, and Y from the reduced image.

[0071] The parameter estimation processing unit 37 compares the second three-primary-color image P3 with the third three-primary-color image P4, calculates the parameters PM of the color conversion matrix multiplied by the color information of the three primary colors in the first three-primary-color image P2, and outputs the parameters PM to the CCM processing unit 23 of the camera system 20.

[0072] At this time, the parameter estimation processing unit 37 calculates the parameter PM that makes the hue and saturation of the first three-primary color image P2 close to the hue and saturation of the second three-primary color image P3. Then, the parameter estimation processing unit 37 replaces and rewrites the parameter a in the formula (2) used by the CCM processing unit 23 with the calculated parameter PM. 00 、a 01 、a 02 、a 10 、a 11 、a 12 、a 20 、a 21 and a 22 The value of .

[0073] The camera system 20 can generate a three-primary color image reflecting the color information of C, M, and Y in the input image P1 by performing CCM processing using formula (2) using the parameter PM. As a result, the image processing apparatus 1 can improve the color reproducibility of the camera system when performing re-mosaic processing.

[0074] [3. First Modification of the Image Processing Device]

[0075] Next, we will refer to Figure 5 and Figure 6 An image processing apparatus according to a first modification example of the embodiment will be described. Figure 5 It is a schematic configuration diagram of an image processing device 1A according to a first modified example of the embodiment.

[0076] Figure 6 1 is a functional block diagram showing a configuration example of an image processing apparatus 1A according to a first modification example of the embodiment.

[0077] like Figure 5 As shown in FIG, the image processing apparatus 1A includes a re-mosaic processing unit 2 and an image processing unit 3A. Figure 1The re-mosaic processing unit 2 described in the figure converts the input image P1 including the colors of R, G, B, C, M and Y into a first three-primary color image P2 of the Bayer array, and outputs the three-primary color image P2 to the camera system 20A.

[0078] The image processing unit 3A generates a second three-primary color image P3 including color information of R, G, B, C, M, and Y instead of the parameter PM from the input image P1, and outputs the second three-primary color image P3 to the camera system 20A. Specifically, as Figure 6 As shown, the image processing unit 3A includes a six-color WB processing unit 31 , a reduction demosaic processing unit 32 , a 3×6 CCM processing unit 35 , and a 2D-LUT processing unit 36 ​​.

[0079] Figure 6 The six-color WB processing unit 31, the reduction demosaic processing unit 32, the 3×6 CCM processing unit 35, and the 2D-LUT processing unit 36 ​​respectively perform the same Figure 2 The six-color WB processing unit 31, the reduction demosaic processing unit 32, the 3×6 CCM processing unit 35, and the 2D-LUT processing unit 36 ​​shown are the same processes.

[0080] On the other hand, Figure 5 As shown, in addition to the Figure 2 In addition to the components in the camera system 20 shown, the camera system 20A according to the first modification further includes an image synthesis unit 25. The image synthesis unit 25 synthesizes the first three primary color image P2 including color information of R, G, and B and not including color information of C, M, and Y input from the gamma correction processing unit 24 with the second three primary color image P3 including color information of R, G, B, C, M, and Y input from the image processing apparatus 1A.

[0081] Therefore, even if no Figure 2 The image processing apparatus 1 can also generate a three-primary color image reflecting the color information of C, M, and Y of the input image P1 using the camera system 20A, by using the 3×3 CCM processing unit 33 , 2D-LUT processing unit 34 , and parameter estimation processing unit 37 shown in FIG.

[0082] [4. Image Processing Device of Modification 2]

[0083] Next, we will refer to Figure 7 An image processing apparatus according to a second modification example of the embodiment will be described. Figure 7 : is a functional block diagram showing a configuration example of an image processing apparatus 1B according to a second modification example of the embodiment.

[0084] like Figure 7As shown, the image processing unit 3B of the image processing apparatus 1B includes a six-color WB processing unit 31 and a machine learning model 38. The six-color WB processing unit 31 and the machine learning model 38 are Figure 2 The machine learning model 38 includes a deep neural network (DNN) or a convolutional neural network (CNN).

[0085] The machine learning model 38 is an arithmetic processing circuit obtained by machine learning so that when the input image P1 is input from the six-color WB processing unit 31, the output is the same as that obtained by Figure 2 The parameter PM output by the parameter estimation processing unit 37 shown is the same parameter PM.

[0086] That is, the machine learning model 38 performs machine learning to calculate parameters PM that make the hue and saturation of the first three primary color image P2 closer to the hue and saturation of the second three primary color image P3 from the input image P1.

[0087] For example, the machine learning model 38 is constructed by repeatedly performing machine learning on teacher data, wherein the input image P1 is used as an input, and when the input image P1 is input, the machine learning model 38 is constructed from the input image P1. Figure 2 The parameters PM output by the image processing unit 3 shown in FIG. 1 are the ground truth while changing the subject of the input image P1.

[0088] Therefore, the image processing device 1B can Figure 2 The hardware configuration of the image processing unit 3 shown in FIG is a simpler hardware configuration of the image processing unit 3B, with Figure 2 Similar to the image processing unit 3 shown in , the parameter PM is calculated from the input image P1 and output to the camera system 20 .

[0089] [5. Modification of Input Image]

[0090] Next, we will refer to Figure 8 A modification example of the input image P1 will be described. Figure 8 : is an explanatory diagram of a modified example of an input image according to an embodiment. Figures 1 to 7 In the embodiment shown in , the input image P1 including color information of R, G, B, C, M, and Y has been described as an example, but the input image according to the embodiment is not limited to the image including color information of R, G, B, C, M, and Y.

[0091] For example, Figure 8As shown in , the input image may be an image including color information of R, G, B, and C but not including color information of M and Y. In addition, the input image may be an image including color information of R, G, and B and color information of at least any one color among color information of C, M, and Y. In addition, the input image may be an image including color information of colors other than C, M, and Y. In this case, for example, instead of formula (3), the image processing device 1 performs CCM processing using a determinant corresponding to the color type included in the input image.

[0092] It should be noted that the effects described in this specification are merely examples and do not limit the disclosure herein, and other effects may also be achieved.

[0093] [6. Appendix]

[0094] It should be noted that the present technology can also have the following configurations. (1)

[0096] An image processing device, comprising:

[0097] a re-mosaic processing unit that converts an array of pixels of red, green, and blue, which are three primary colors of light, in an input image, the input image including pixels of the three primary colors and pixels of colors other than the three primary colors, into a Bayer array to generate a first three-primary color image including color information of the three primary colors and excluding color information of colors other than the three primary colors; and

[0098] An image processing unit generates a second three-primary color image from the input image, the second three-primary color image including color information of the three primary colors and color information other than the three primary colors. (2)

[0100] The image processing apparatus according to (1), wherein

[0101] Re-mosaic processing unit

[0102] In a subsequent stage, the first three primary color image is output to a camera system, and

[0103] Image processing unit

[0104] generating a third three-primary color image including color information of the three primary colors and not including color information other than the three primary colors from the input image, and

[0105] The second three-primary-color image and the third three-primary-color image are compared, coefficients of a color conversion matrix multiplied by color information of the three primary colors in the first three-primary-color image are calculated, and the coefficients are output to the camera system. (3)

[0107] The image processing device according to (2), wherein

[0108] Image processing unit

[0109] Coefficients for making the hue and saturation of the first three-primary-color image closer to the hue and saturation of the second three-primary-color image are calculated. (4)

[0111] The image processing apparatus according to (1), wherein

[0112] Re-mosaic processing unit

[0113] In a subsequent stage, the first three primary color image is output to a camera system, and

[0114] Image processing unit

[0115] The second three-primary color image is output to the camera system. (5)

[0117] The image processing device according to any one of (1) to (4), wherein

[0118] Image processing unit

[0119] A reduced image of the input image is generated by dividing the input image into multiple blocks and averaging and aggregating color information of multiple pixels of the same color into one pixel for the three primary colors included in the blocks and each color other than the three primary colors, and a second three-primary color image is generated from the reduced image. (6)

[0121] The image processing device according to (2) or (3), wherein

[0122] Image processing unit

[0123] A reduced image of the input image is generated by dividing the input image into multiple blocks and, for each of the three primary colors included in the blocks, averaging and aggregating color information of multiple pixels of the same color into one pixel, and a third three-primary color image is generated from the reduced image. (7)

[0125] The image processing apparatus according to (1), wherein

[0126] The image processing unit includes:

[0127] A machine learning model obtained through machine learning calculates, based on an input image, coefficients that make the hue and saturation of the first three primary color image closer to the hue and saturation of the second three primary color image as coefficients of a color conversion matrix to be multiplied by the color information of the three primary colors in the first three primary color image. (8)

[0129] An image processing method performed by an image processing device, the image processing method comprising:

[0130] converting an array of pixels of red, green, and blue, which are three primary colors of light, in an input image including pixels of the three primary colors and pixels of colors other than the three primary colors into a Bayer array to generate a first three-primary color image including color information of the three primary colors and excluding color information of colors other than the three primary colors; and

[0131] A second three-primary color image is generated from the input image, the second three-primary color image including color information of the three primary colors and color information other than the three primary colors.

[0132] Reference Number List

[0133] 1. 1A, 1B image processing device

[0134] 10 Imaging Unit

[0135] 2 Re-mosaic processing unit

[0136] 3. 3A and 3B image processing units

[0137] 31 six-color WB processing unit

[0138] 32 Reduced Demosaic Processing Units

[0139] 33 3×3 CCM processing units

[0140] 34, 36 2D-LUT processing units

[0141] 35 3×6 CCM processing units

[0142] 37 Parameter estimation processing unit

[0143] 38 Machine Learning Models

[0144] 20, 20A camera system

[0145] 21 De-mosaicing Unit

[0146] 22WB processing unit

[0147] 23CCM processing unit

[0148] 24 gamma correction processing units

[0149] 25 Image Synthesis Unit

[0150] P1 input image

[0151] P2 first three primary color image

[0152] P3 Second and third primary color images

[0153] P4 third primary color image

[0154] PM parameters

Claims

1. An image processing device, comprising: a re-mosaic processing unit that converts an array of pixels of red, green, and blue, which are three primary colors of light, in an input image, the input image including pixels of the three primary colors and pixels of colors other than the three primary colors, into a Bayer array to generate a first three-primary color image including color information of the three primary colors and excluding color information of colors other than the three primary colors; as well as An image processing unit generates a second three-primary color image from the input image, the second three-primary color image including color information of the three primary colors and color information other than the three primary colors.

2. The image processing apparatus according to claim 1, wherein The re-mosaic processing unit In the subsequent stage, the first three primary color images are output to the camera system and the image processing unit generating a third three-primary color image including color information of the three primary colors and not including color information other than the three primary colors from the input image, and The second three-primary-color image and the third three-primary-color image are compared, coefficients of a color conversion matrix multiplied by color information of three primary colors in the first three-primary-color image are calculated, and the coefficients are output to the camera system.

3. The image processing apparatus according to claim 2, wherein The image processing unit The coefficients for making the hue and saturation of the first three-primary-color image closer to the hue and saturation of the second three-primary-color image are calculated. The image processing apparatus according to claim 1 , wherein Re-mosaic processing unit In the subsequent stage, the first three primary color images are output to the camera system and the image processing unit The second three-primary color image is output to the camera system.

5. The image processing apparatus according to claim 1, wherein The image processing unit A reduced image of the input image is generated by dividing the input image into multiple blocks and averaging and aggregating color information of multiple pixels of the same color into one pixel for the three primary colors included in the blocks and each color other than the three primary colors, and a second three-primary color image is generated from the reduced image. The image processing apparatus according to claim 2 , wherein The image processing unit A reduced image of the input image is generated by dividing the input image into multiple blocks and, for each of the three primary colors included in the blocks, averaging color information of multiple pixels of the same color and aggregating them into one pixel, and a third three-primary color image is generated from the reduced image.

7. The image processing apparatus according to claim 1, wherein The image processing unit includes: A machine learning model obtained through machine learning calculates, based on the input image, coefficients that make the hue and saturation of the first three-primary-color image closer to the hue and saturation of the second three-primary-color image as coefficients of a color conversion matrix to be multiplied by the color information of the three primary colors in the first three-primary-color image.

8. An image processing method performed by an image processing device, the image processing method comprising: converting an array of pixels of red, green, and blue, which are three primary colors of light, in an input image including pixels of the three primary colors and pixels of colors other than the three primary colors into a Bayer array to generate a first three-primary color image including color information of the three primary colors and excluding color information of colors other than the three primary colors; as well as A second three-primary color image is generated from the input image, the second three-primary color image including color information of the three primary colors and color information other than the three primary colors.

Images