Image processing apparatus, image processing method, program, and imaging apparatus

Abstract

This disclosure relates to an image processing apparatus, an image processing method, a program, and an imaging apparatus that can suppress the degradation of image quality. The image processing apparatus includes a demosaicing unit that performs demosaicing on an image generated by an image sensor having a first pixel array arranged in an oblique grid pattern, the first pixel receiving light of a first color component consisting of red, green, and blue or yellow color components; a second pixel receiving light of a second color component consisting of red color components; a third pixel receiving light of a third color component consisting of green color components; and a fourth pixel receiving light of a fourth color component consisting of blue color components; or a second pixel array arranged in an oblique grid pattern, the first pixel, second pixel, third pixel, fourth pixel, and a fifth pixel receiving light of a fifth color component different from the first to fourth color components. The technology of this disclosure can be applied, for example, to an image processing apparatus that performs demosaicing.

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus, an image processing method, a program, and an imaging apparatus, and more particularly to an image processing apparatus, an image processing method, a program, and an imaging apparatus that can suppress the deterioration of image quality. [Background technology]

[0002] To make the camera smaller and lighter, a single-chip camera using one image sensor is employed instead of a three-chip camera using three image sensors.

[0003] A color coding filter, consisting of a color filter array assigned to each pixel, is installed on the light incident surface of the image sensor of a single-chip camera. This allows the image sensor to generate a signal of the color components color-coded by the color coding filter for each pixel. An example of a color filter array constituting the color coding filter is a three-primary-color filter array of R (red), G (green), and B (blue).

[0004] A single-chip camera obtains a signal for one color component per pixel using an image sensor, and generates color signals other than those already present in each pixel through linear interpolation. As a result, a single-chip camera can obtain an image similar to that obtained by a three-chip camera.

[0005] A Bayer array of three primary color filters is often used as the color filter array that makes up a color coding filter. In a Bayer array, the G color filter is arranged in a checkerboard pattern, and R and B are arranged alternately in the remaining space, one row at a time.

[0006] Each pixel in a Bayer array image sensor is fitted with a filter of one of the three primary colors: R, G, or B. Each pixel outputs only the image signal corresponding to the color of the filter it is fitted with. A pixel fitted with an R color filter outputs the R component of the image signal, but not the G component or B component. Similarly, a pixel fitted with a G component outputs only the G component of the image signal, but not the R or B component. A pixel fitted with a B component outputs only the B component of the image signal, but not the R or G component.

[0007] Patent Document 1 discloses a technique for interpolating missing color components. In Patent Document 1, first, the interpolated color difference of each pixel position in the input image from the image sensor is determined in two directions, horizontal and vertical, and statistical quantities of the five pixels in the vicinity of the pixel of interest are calculated. Using the calculated statistical quantities, the orientation (horizontal or vertical) of the pixel of interest is detected, and according to the detection result, the horizontal color difference and vertical color difference are mixed (proportionally) to determine the color difference (RG, BG). Based on the color difference (RG, BG) determined in this way, the image signal of the G pixel, the image signal of the R pixel, and the image signal of the B pixel are determined, respectively, and the output image is generated.

[0008] Furthermore, Patent Document 1 discloses a technique for obtaining an image signal equivalent to the CCD output of a three-chip camera using the CCD image sensor of a single-chip camera. In the technique of Patent Document 1, the pixel value of the pixel of interest in the output image is predicted by a sum-of-products operation using coefficients obtained in advance through learning, with the pixel values ​​of the pixel of interest and its surrounding pixels in the input image as variables.

[0009] In single-chip camera image sensors, light is generally allowed to pass through an optical low-pass filter before being incident on the image sensor to avoid the effects of false colors and artifacts. However, passing the light through an optical low-pass filter in this way can sometimes cause image blurring. Patent Document 1 discloses a technique to suppress such image quality degradation and obtain image signals for each color component. Specifically, a specified area is selected from a first image output from a single-chip pixel unit, representative values ​​for each color component of that specified area are calculated, and pre-stored coefficients are read out based on feature quantities obtained from the pixel values ​​of the specified area. Then, predetermined pixels within the specified area are designated as filter taps, and the pixel values ​​of each color component of the filter tap are converted to conversion values. Using the conversion values ​​as variables, the pixel values ​​of each color component of the image are calculated by sum-of-products operations using the read-out coefficients.

[0010] Furthermore, Patent Document 1 discloses a technique for reducing false color and noise in image signals acquired by a single-chip camera. Specifically, a reference area is set from a first image composed of image signals output from a single-chip pixel section with a Bayer array, and by changing the area of ​​the reference area and evaluating statistical quantities obtained from the pixel values ​​of pixels within the reference area, the directionality of the point of interest in the first image is detected, thereby detecting the directionality of the point of interest with higher accuracy. As a result, even when an input image containing high-frequency components close to the Nyquist frequency is input, the directionality can be accurately detected by expanding the reference area, and false color and noise can be reliably reduced.

[0011] Patent Document 2 discloses a zigzag pixel arrangement in which chromatic pixels and high-sensitivity pixels are arranged in a checkerboard pattern. [Prior art documents] [Patent Documents]

[0012] [Patent Document 1] Patent No. 6435560 [Patent Document 2] Japanese Patent Publication No. 2007-288394 [Overview of the project] [Problems that the invention aims to solve]

[0013] Current demosaicing processes may not be able to minimize image quality degradation, and a new demosaicing process that can reduce image quality degradation is needed.

[0014] This technology was developed in light of these circumstances, and aims to realize an image processing device, image processing method, program, and imaging device that can suppress the degradation of image quality. [Means for solving the problem]

[0015] The image processing apparatus in the first aspect of this disclosure is an image processing apparatus having a first pixel array in which a first pixel that receives light of a first color component consisting of a color component including red, green, and blue or a color component of yellow, a second pixel that receives light of a second color component consisting of a color component of red, a third pixel that receives light of a third color component consisting of a color component of green, and a fourth pixel that receives light of a fourth color component consisting of a color component of blue are arranged in an oblique grid, or a first pixel array having the first pixel, the second pixel, the third pixel, the fourth pixel, and a fifth pixel that receives light of a fifth color component different from the first to the fourth color component arranged in an oblique grid. The system includes a demosaicing unit that performs demosaicing on an image generated by an image sensor having a 2-pixel array, the demosaicing unit includes a first-color component interpolation generation unit that generates a first-color component interpolation pixel value for each pixel of the image other than the first pixel to interpolate the first color component, a second-color component virtual color difference generation unit that generates a virtual color difference for the second color component based on the difference between the pixel value of the second pixel and the first-color component interpolation pixel value for the second pixel, and a third-color component virtual color difference generation unit that generates a virtual color difference for the third color component based on the difference between the pixel value of the third pixel and the first-color component interpolation pixel value for the third pixel A third color component virtual color difference generation unit generates a virtual color difference for the fourth color component based on the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component for the fourth pixel. A fifth color component virtual color difference generation unit generates a virtual color difference for the fifth color component based on the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component for the fifth pixel. A first color component virtual color difference generation unit has a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix using the pixel value of the first pixel, and each pixel has the pixel value of the first color component. A first color component image generation unit generates a color component image; a second color difference component image generation unit generates a second color difference component image with the same arrangement and resolution as the first color component image using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel; a third color difference component image generation unit generates a third color difference component image with the same arrangement and resolution as the first color component image using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel; and a fourth color difference component generates a second color difference component image with the same arrangement and resolution as the first color component image using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel.The system includes: a fourth color difference component image generation unit that generates a fourth color difference component image having the same arrangement and resolution as the first color component image; a fifth color difference component image generation unit that generates a fifth color difference component image having the same arrangement and resolution as the first color component image using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel; a first adder that generates a second color component image having the same arrangement and resolution as the first color component image by adding the second color difference component image and the first color component image; a second adder that generates a third color component image having the same arrangement and resolution as the first color component image by adding the third color difference component image and the first color component image; a third adder that generates a fourth color component image having the same arrangement and resolution as the first color component image by adding the fourth color difference component image and the first color component image; and a fourth adder that generates a fifth color component image having the same arrangement and resolution as the first color component image by adding the fifth color difference component image and the first color component image. ,

[0016] The second aspect of the present disclosure is an image processing method comprising a first pixel array in which a first pixel that receives light of a first color component consisting of a color component including red, green, and blue or a color component of yellow, a second pixel that receives light of a second color component consisting of a color component of red, a third pixel that receives light of a third color component consisting of a color component of green, and a fourth pixel that receives light of a fourth color component consisting of a color component of blue are arranged in an oblique grid, or a second pixel array in which the first pixel, the second pixel, the third pixel, the fourth pixel, and a fifth pixel that receives light of a fifth color component different from the first to the fourth color component are arranged in an oblique grid. The process includes demosaicing an image generated by an image sensor having a pixel array, wherein the demosaicing process involves generating a first color component interpolation pixel value for each pixel of the image other than the first pixel to interpolate the first color component; generating a virtual color difference for the second color component for the second pixel based on the difference between the pixel value of the second pixel and the first color component interpolation pixel value; generating a virtual color difference for the third color component for the third pixel based on the difference between the pixel value of the third pixel and the first color component interpolation pixel value; and for the fourth pixel, the fourth The process involves generating a virtual color difference for the fourth color component based on the difference between the pixel value of a pixel and the interpolated pixel value of the first color component; generating a virtual color difference for the fifth color component based on the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component; generating a first color component image using the pixel value of the first pixel, having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component; and using the virtual color difference for the second color component, the pixel value of the first pixel, and the pixel value of the second pixel Then, generate a second color difference component image with the same arrangement and resolution as the first color component image; generate a third color difference component image with the same arrangement and resolution as the first color component image using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel; generate a fourth color difference component image with the same arrangement and resolution as the first color component image using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel; and use the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel,This includes generating a fifth color difference component image having the same arrangement and resolution as the first color component image; generating a second color component image having the same arrangement and resolution as the first color component image by adding the second color difference component image and the first color component image; generating a third color component image having the same arrangement and resolution as the first color component image by adding the third color difference component image and the first color component image; generating a fourth color component image having the same arrangement and resolution as the first color component image by adding the fourth color difference component image and the first color component image; and generating a fifth color component image having the same arrangement and resolution as the first color component image by adding the fifth color difference component image and the first color component image.

[0017] The program of the third aspect of this disclosure provides a computer with a first pixel array in which a first pixel receiving light of a first color component consisting of a color component including red, green, and blue or a color component of yellow, a second pixel receiving light of a second color component consisting of a color component of red, a third pixel receiving light of a third color component consisting of a color component of green, and a fourth pixel receiving light of a fourth color component consisting of a color component of blue, arranged in an oblique grid, or the first pixel, the second pixel, the third pixel, the fourth pixel, and a fifth pixel receiving light of a fifth color component different from the first to the fourth color component arranged in an oblique grid. The process includes demosaicing of an image generated by an image sensor having an arrayed second pixel array, wherein the demosaicing process involves generating a first color component interpolation pixel value for each pixel of the image other than the first pixel to interpolate the first color component; generating a virtual color difference for the second color component for the second pixel based on the difference between the pixel value of the second pixel and the first color component interpolation pixel value; generating a virtual color difference for the third color component for the third pixel based on the difference between the pixel value of the third pixel and the first color component interpolation pixel value; and for the fourth pixel, The process involves generating a virtual color difference for the fourth color component based on the difference between the pixel values ​​of four pixels and the interpolated pixel values ​​of the first color component; generating a virtual color difference for the fifth color component based on the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component; generating a first color component image using the pixel value of the first pixel, having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component; and using the virtual color difference for the second color component, the pixel value of the first pixel, and the pixel value of the second pixel. Then, a second color difference component image having the same arrangement and resolution as the first color component image is generated; a third color difference component image having the same arrangement and resolution as the first color component image is generated using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel; a fourth color difference component image having the same arrangement and resolution as the first color component image is generated using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel; and a fifth color difference component image having the same arrangement and resolution as the first color component image is generated using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel.This process involves generating a fifth color difference component image with the same arrangement and resolution as the first color component image; generating a second color component image with the same arrangement and resolution as the first color component image by adding the second color difference component image and the first color component image; generating a third color component image with the same arrangement and resolution as the first color component image by adding the third color difference component image and the first color component image; generating a fourth color component image with the same arrangement and resolution as the first color component image by adding the fourth color difference component image and the first color component image; and generating a fifth color component image with the same arrangement and resolution as the first color component image by adding the fifth color difference component image and the first color component image.

[0018] The imaging device according to the fourth aspect of this disclosure comprises the image sensor described in the first aspect and the image processing device described in the first aspect.

[0019] In the first to fourth aspects of this disclosure, demosaicing is performed on an image sensor having a first pixel array in which a first pixel that receives light of a first color component consisting of a color component including red, green, and blue or a color component of yellow is arranged in an oblique grid, a second pixel that receives light of a second color component consisting of a color component of red, a third pixel that receives light of a third color component consisting of a color component of green, and a fourth pixel that receives light of a fourth color component consisting of a color component of blue, or a second pixel array in which the first pixel, the second pixel, the third pixel, the fourth pixel, and a fifth pixel that receives light of a fifth color component different from the first to the fourth color component are arranged in an oblique grid. In the demosaicing process, for each pixel of the image other than the first pixel, a first color component interpolation pixel value is generated to interpolate the first color component; for the second pixel, a virtual color difference of the second color component is generated by the difference between the pixel value of the second pixel and the first color component interpolation pixel value; for the third pixel, a virtual color difference of the third color component is generated by the difference between the pixel value of the third pixel and the first color component interpolation pixel value; for the fourth pixel, a virtual color difference of the fourth color component is generated by the difference between the pixel value of the fourth pixel and the first color component interpolation pixel value; for the fifth pixel, a virtual color difference of the fifth color component is generated by the difference between the pixel value of the fifth pixel and the first color component interpolation pixel value; and using the pixel value of the first pixel, the horizontal and vertical directions of the pixel array are processed. Having a pixel array in which pixels in a rectangular shape parallel to each other are arranged in a matrix, a first color component image is generated in which each pixel has the pixel value of the first color component; a second color difference component image having the same array and resolution as the first color component image is generated using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel; a third color difference component image having the same array and resolution as the first color component image is generated using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel; a fourth color difference component image having the same array and resolution as the first color component image is generated using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel; and a fifth color difference component is generated using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel.The process involves generating a fifth color difference component image with the same arrangement and resolution as the first color component image, adding the second color difference component image and the first color component image to generate a second color component image with the same arrangement and resolution as the first color component image, adding the third color difference component image and the first color component image to generate a third color component image with the same arrangement and resolution as the first color component image, adding the fourth color difference component image and the first color component image to generate a fourth color component image with the same arrangement and resolution as the first color component image, and adding the fifth color difference component image and the first color component image to generate a fifth color component image with the same arrangement and resolution as the first color component image.

[0020] The image processing device and imaging device may be independent devices or modules incorporated into other devices. [Effects of the Invention]

[0021] This technology makes it possible to suppress the degradation of image quality.

[0022] The effects described herein are not necessarily limited to those described herein and may include any of the effects described herein. [Brief explanation of the drawing]

[0023] [Figure 1] This is a block diagram showing an example of the configuration of an imaging device according to one embodiment of the present disclosure. [Figure 2] This is a diagram showing the first pixel array of the image sensor. [Figure 3] This figure shows the second pixel array of the image sensor. [Figure 4] This figure shows an example of the first demosaicing process performed by the demosaicing unit. [Figure 5] This figure shows the correspondence between the input pixels and generated pixels in the demosaicing section for the WRGB array. [Figure 6] This figure shows the correspondence between the input pixels and generated pixels in the demosaicing region for the WRGAB sequence. [Figure 7] This figure shows a second example of demosaicing by the demosaicing unit. [Figure 8] This figure shows the correspondence between the input pixels and generated pixels in the demosaicing region for the WRGAB sequence. [Figure 9] This block diagram shows an example of the configuration of the first example of the demosaicing section. [Figure 10] This block diagram shows an example of the second configuration of the demosaicing section. [Figure 11] This is a block diagram showing a detailed configuration example of the interpolation W' generation unit. [Figure 12] This diagram shows an example of a W' code tap. [Figure 13] This figure shows an example of a W' filter tap. [Figure 14] This block diagram shows a detailed configuration example of the W' code calculation unit. [Figure 15] This diagram illustrates an example of 1-bit DR quantization. [Figure 16] This is a block diagram showing a detailed configuration example of the high-resolution W generation unit. [Figure 17] This diagram shows an example of a W-code tap. [Figure 18] This figure shows an example of a W filter tap. [Figure 19] This is a block diagram showing a detailed configuration example of the high-resolution (RW) generation unit. [Figure 20] This figure shows an example of the dynamic range calculation range. [Figure 21] This figure shows an example of a (RW) cord tap. [Figure 22] This figure shows an example of a (RW) filter tap. [Figure 23] This diagram shows an example of a (GW) code tap. [Figure 24] (GW) This is a diagram showing an example of a filter tap. [Figure 25] This figure shows an example of a (BW) code tap. [Figure 26] This figure shows an example of a (BW) filter tap. [Figure 27] This diagram shows an example of (AW) code tapping. [Figure 28] This figure shows an example of an (AW) filter tap. [Figure 29] This is a flowchart explaining the demosaicing process. [Figure 30] This is a block diagram showing an example configuration of a learning device for learning the W' coefficient. [Figure 31] This is a diagram illustrating the student and teacher images generated from a high-resolution image W. [Figure 32] This figure shows a detailed configuration example of the W' coefficient data generation unit. [Figure 33] This is a flowchart explaining the learning process for learning the W' coefficient. [Figure 34] This block diagram shows an example configuration of a learning device corresponding to the first example configuration of the demosaicing section. [Figure 35] This figure illustrates student and teacher images generated from high-resolution images of the W, R, G, and B components. [Figure 36] This block diagram shows an example configuration of a learning device corresponding to the second example configuration of the demosaicing section. [Figure 37] This figure illustrates student and teacher images generated from high-resolution images of the W, R, G, B, and A components. [Figure 38] This figure shows a detailed configuration example of the W coefficient data generation unit. [Figure 39] This figure shows an example of the configuration of the (RW) coefficient data generation unit. [Figure 40] This flowchart explains the learning process for learning the W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient. [Figure 41] This figure shows an example of the optical system configuration shown in Figure 1. [Figure 42] This figure shows an example of an optical low-pass filter configuration. [Figure 43] This is a diagram illustrating a birefringent optical element. [Figure 44] This is a diagram illustrating a four-layer birefringent optical element. [Figure 45] This figure shows the relationship between the light-gathering area of ​​the on-chip microlens and the light-gathering area of ​​the generated pixels, depending on the presence or absence of an optical low-pass filter. [Figure 46] This figure shows the spectral sensitivity characteristics of the image sensor. [Figure 47] This figure shows a first example configuration of a photodiode and on-chip microlens for an image sensor when the pixel array is a WRGAB array. [Figure 48] This figure shows the exposure saturation time for each pixel in the pixel array shown in Figure 47. [Figure 49] This figure shows a second example configuration of the photodiode and on-chip microlens of an image sensor when the pixel array is a WRGAB array. [Figure 50] This figure shows the exposure saturation time for each pixel in the pixel array shown in Figure 49. [Figure 51] This figure shows a third example configuration of the photodiode and on-chip microlens of an image sensor when the pixel array is a WRGAB array. [Figure 52] This figure shows examples of RAW data and image formats after demosaicing and color correction. [Figure 53] This figure shows examples of RAW data when the W pixel is divided into four parts, and the image format after demosaicing and color correction. [Figure 54] This figure shows examples of RAW data when the W pixel is divided into four parts, and the image format after demosaicing and color correction. [Figure 55] This is a block diagram showing an example configuration of one embodiment of a computer to which the technology of this disclosure is applied. [Modes for carrying out the invention]

[0024] Hereinafter, embodiments for carrying out the technology of this disclosure (hereinafter referred to as "embodiments") will be described with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals to avoid redundant explanations. The explanation will be given in the following order. 1. Example of imaging device configuration 2. Example of Demosaicing Configuration 3. Detailed Configuration Example of the Interpolation W' Generation Unit 4. Detailed Configuration Example of the High-Resolution W Generation Unit 5. Detailed Configuration Example of the High Resolution (RW) Generation Unit 6. Detailed Configuration Example of the High-Resolution (GW) Generation Unit 7. Detailed Configuration Example of the High-Resolution (BW) Generation Unit 8. Detailed Configuration Example of the High Resolution (AW) Generation Unit 9. Flowchart for Demosaicing 10. Learning Principles of Tap Coefficients 11. Example configuration of a learning device for the W' coefficient 12. Flowchart for learning the W' coefficient 13. Example configuration of a learning device corresponding to the first configuration example (WRGB array) 14. Example configuration of a learning device corresponding to the second configuration example (WRGAB array) 15. Learning Process Flowchart for W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient 16. Example of an optical system configuration 17. Example of pixel size configuration 18. Color correction processing in the color processing section. 19. Example Data Format 20. Computer Configuration Examples

[0025] <1. Example of imaging device configuration> Figure 1 is a block diagram showing an example configuration of an imaging device, which is one embodiment of the present disclosure.

[0026] The imaging device 10 shown in Figure 1 is a device that captures images of a subject and generates captured images. The captured images may be either still images or moving images. The imaging device 10 comprises an optical system 11, an image sensor 12, an analog signal processing unit 13, an A / D color separation unit 14, a defect correction unit 15, a white balance unit 16, a demosaicing unit 17, a color processing unit 18, a Y color difference conversion unit 19, a codec unit 20, an output unit 21, and a control unit 22. The block from the image sensor 12 to the defect correction unit 15 constitutes the imaging system 23, and the block from the white balance unit 16 to the codec unit 20 constitutes the image processing unit 24. The image processing unit 24 may be provided as an image processing device separate from the imaging device 10.

[0027] The optical system 11 includes, for example, a zoom lens (not shown), a focus lens, an aperture, an optical low-pass filter, an IR cut filter, etc. The optical system 11 causes external subject light to enter the image sensor 12.

[0028] The image sensor 12 is composed of a CMOS (Complementary Metal Oxide Semiconductor) image sensor having a pixel array in which multiple pixels (photodetectors) with different wavelength bands of light received (hereinafter referred to as color components as appropriate) are arranged in a predetermined sequence. The image sensor 12 receives incident light from the optical system 11, performs photoelectric conversion for each pixel, and outputs an image signal as an electrical signal corresponding to the incident light from the optical system 11. The pixel array of the image sensor 12 can take the following first pixel arrangement or second pixel arrangement.

[0029] Figure 2 shows the possible first pixel arrangements of the pixel array of the image sensor 12.

[0030] As shown in Figure 2, the image sensor 12 has multiple pixels arranged in a diagonal grid pattern. In Figure 2, the dotted lines of the cross represent the horizontal and vertical directions of the pixel array, and the diagonal grid pattern means that the pixels are arranged at a 45-degree angle to the horizontal and vertical directions of the pixel array. In other words, in a diagonal grid pixel arrangement, the direction of pixel arrangement is at a 45-degree angle to the horizontal and vertical directions of the pixel array. In a diagonal grid pixel arrangement, one rectangle tilted at a 45-degree angle represents one pixel. Hereafter, a pixel tilted at a 45-degree angle will also be referred to as a rhombic pixel.

[0031] The first pixel array is composed of diamond-shaped pixels arranged in a diagonal grid. In the first pixel array, W pixels, R pixels, G pixels, and B pixels are arranged in a predetermined configuration. More specifically, W pixels tilted at a 45-degree angle are arranged continuously in the horizontal and vertical directions, with R pixels, G pixels, or B pixels placed between adjacent W pixels. While the W pixels are arranged continuously in the horizontal and vertical directions, the R pixels, G pixels, and B pixels are arranged alternately in the horizontal and vertical directions. For a 2x2 W pixel array, a 2x2 pixel array shifted by half the pixel size pitch in the tilt direction by 45 degrees contains R pixels, G pixels, G pixels, and B pixels. Therefore, in the first pixel array, W pixels are the most numerous and densest, followed by G pixels, and R and B pixels are the least numerous.

[0032] A W pixel is a pixel equipped with a color filter that transmits all color components (first color component) including red (R), green (G), and blue (B). The wavelength range of light received by a W pixel is, for example, about 400 to 1000 nm. Such pixels are also called white pixels, and for convenience of explanation, they are sometimes described as white (W) in correspondence with red (R), green (G), and blue (B). An R pixel is a pixel equipped with a color filter that transmits the red (R) color component (second color component). The wavelength range of light received by an R pixel is, for example, about 600 to 700 nm. A G pixel is a pixel equipped with a color filter that transmits the green (G) color component (third color component). The wavelength range of light received by a G pixel is, for example, about 500 to 600 nm. A B pixel is a pixel equipped with a color filter that transmits the blue (B) color component (fourth color component). The wavelength range of light received by a B pixel is, for example, around 400-500 nm. A W pixel provides the W component of the image signal. An R pixel provides the R component of the image signal, but not the G component or the B component. Similarly, a G pixel provides the G component of the image signal, but not the R component or the B component. A B pixel provides the B component of the image signal, but not the R component or the G component.

[0033] Figure 3 shows the possible second pixel arrangements of the pixel array of the image sensor 12.

[0034] The second pixel array arranges diamond-shaped pixels at a 45-degree angle, and the arrangement of W pixels is the same as that of the first pixel array shown in Figure 2. In Figure 3, the dotted lines of the cross represent the horizontal and vertical directions of the pixel array, and each rectangle arranged in a diagonal grid represents one pixel.

[0035] In the second pixel array, the arrangement of pixels placed between W pixels differs from that of the first pixel array. In the first pixel array, R pixels, G pixels, G pixels, and B pixels were arranged in a 2x2 pixel arrangement that was shifted by half a pitch by 45 degrees in the direction of the 2x2 W pixels. In contrast, in the second pixel array, R pixels, G pixels, A pixels, and B pixels are arranged. In other words, one of the two G pixels in the first pixel array is replaced with an A pixel in the second pixel array. Here, an A pixel is a pixel on which a color filter is placed that transmits a color component (fifth color component) that is different from the first color component (W component) to the fourth color component (B component). Here, a color component that is different from the first color component (W component) to the fourth color component (B component) means that its transmitted wavelength band does not perfectly match any of the first to fourth color components, and it is also possible that a part of its transmitted wavelength band overlaps with any of the first to fourth color components. A pixel A is a pixel equipped with a color filter that transmits a color component (the fifth color component) corresponding to, for example, infrared (IR), cyan (C), emerald green (EG), orange (O), violet (V), or ultraviolet (UV). When a pixel A receives the infrared (IR) color component, the wavelength range of the light received by the pixel A is, for example, about 800 to 1000 nm. When a pixel A receives the cyan (C) color component, the wavelength range of the light received by the pixel A is, for example, about 490 to 560 nm. When a pixel A receives the emerald green (EG) color component, the wavelength range of the light received by the pixel A is, for example, about 450 to 550 nm. When a pixel A receives the orange (O) color component, the wavelength range of the light received by the pixel A is, for example, about 590 to 640 nm. When a pixel A receives the violet (V) color component, the wavelength range of the light received by the pixel A is, for example, approximately 380-430 nm. When a pixel A receives the ultraviolet (UV) color component, the wavelength range of the light received by the pixel A is, for example, approximately 200-400 nm. An image signal of component A is obtained from a pixel A, but image signals of components R, G, and B are not obtained.

[0036] In the second pixel array, W pixels are the most numerous and densely arranged, while R, G, B, and A pixels are arranged in equal numbers but fewer than W pixels.

[0037] If we define W pixels as achromatic pixels and R, G, B, and A pixels as chromatic pixels, then the first and second pixel arrays are zigzag pixel arrays in which chromatic and achromatic pixels are arranged in a checkerboard pattern. W pixels are arranged more densely than R, G, B, and A pixels and serve as the reference pixels in the demosaicing process performed in the demosaicing section 17 described later. In the following, for ease of understanding, the first pixel array may be referred to as the WRGB array and the second pixel array as the WRGAB array.

[0038] In the following explanation, a W pixel is described as a pixel that receives light containing all three color components (first color components): red (R), green (G), and blue (B). However, a W pixel may also be a pixel equipped with a color filter that transmits yellow (Y) color components. In other words, a W pixel is composed of a pixel that receives light consisting of either red (R), green (G), and blue (B) color components as the first color component, or yellow (Y) color components. The wavelength range of the yellow (Y) color component is, for example, approximately 550-590 nm.

[0039] Returning to the explanation of Figure 1, the analog signal processing unit 13 performs predetermined analog signal processing, such as correlated double sampling, on the image signal output from the image sensor 12. The A / D color separation unit 14 performs A / D conversion (analog / digital conversion) on the image signal output from the analog signal processing unit 13, and further performs color separation processing to output image data for white (W), red (R), green (G), blue (B), or other (A). The defect correction unit 15 performs defect correction processing on the WRGB array or WRGAB array image data output from the A / D color separation unit 14.

[0040] The white balance unit 16 performs white balance processing on the WRGB or WRGAB sequence image data output from the defect correction unit 15.

[0041] The demosaicing unit 17 performs demosaicing on the WRGB or WRGAB array image data output from the white balance unit 16. If the pixel array is a WRGB array, the demosaicing unit 17 generates W, R, G, and B images through demosaicing and outputs them to the color processing unit 18. If the pixel array is a WRGAB array, the demosaicing unit 17 generates W, R, G, A, and B images through demosaicing and outputs them to the color processing unit 18.

[0042] Figure 4 shows an example of the first demosaicing process performed by the demosaicing unit 17 when the pixel array is a WRGB array.

[0043] As shown in Figure 4, the demosaicing unit 17 generates W, R, G, and B images from the WRGB array image data of diamond-shaped pixels, by arranging multiple rectangular pixels parallel to the horizontal and vertical directions of the pixel array. At this time, the resolution (number of pixels) of each of the W, R, G, and B images (collectively referred to as the WRGB image) is the same as the resolution of the W pixel.

[0044] Figure 5 shows the correspondence between the input pixels and generated pixels of the demosaicing unit 17 in the case of a WRGB array. In Figure 5, the pixels of the generated WRGB image (generated pixels) for the arrangement of W pixels, R pixels, G pixels, and B pixels are shown by dashed lines.

[0045] As shown in Figure 5, the generated pixels correspond to the regions surrounding the four corners of the diamond shape of the W pixels in the horizontal and vertical directions. The number of generated pixels in the W image is the same as the number of W pixels, and the resolution of the W image is equal to the resolution of the W pixels in the pixel array. The resolution of the R image is four times that of the R pixels. The resolution of the G image is twice that of the G pixels. The resolution of the B image is four times that of the B pixels.

[0046] In the case of a WRGAB array, although not shown in the diagram, the demosaicing unit 17 generates W, R, G, A, and B images from the WRGB array image data of diamond-shaped pixels by arranging multiple rectangular pixels parallel to the horizontal and vertical directions of the pixel array.

[0047] Figure 6 shows the correspondence between the input pixels and generated pixels of the demosaicing section 17 in the case of the WRGAB array. In Figure 6, the pixels (generated pixels) of the generated W, R, G, A, and B images (collectively referred to as the WRGAB image) are shown by dashed lines, corresponding to the arrangement of W, R, G, A, and B pixels.

[0048] As shown in Figure 6, the resolution of the W image is equal to the resolution of the W pixel in the pixel array. The resolution of the R image is four times that of the R pixel. The resolution of the G image is four times that of the G pixel. The resolution of the A image is four times that of the A pixel. The resolution of the B image is four times that of the B pixel.

[0049] In the examples shown in Figures 4 to 6, the resolution of pixels other than the W pixel is improved, but the resolution of the W image remains unchanged. The demosaicing unit 17 can also perform demosaicing to improve the resolution of the W image and generate a WRGB or WRGAB image.

[0050] Figure 7 shows a second demosaicing example by the demosaicing unit 17, which shows an example where the resolution of the W image has been improved. In Figure 7, the pixel array arrangement is the same WRGB arrangement as in Figure 4.

[0051] As shown in Figure 7, the demosaicing unit 17 generates a WRGB image from the WRGB array image data of diamond-shaped pixels by arranging multiple rectangular pixels parallel to the horizontal and vertical directions of the pixel array. At this time, the resolution of the WRGB image is four times the resolution of the W pixels.

[0052] Figure 8 shows the correspondence between the input pixels and generated pixels of the demosaicing section 17 in the case of a WRGAB array. In Figure 8, the pixels of the generated WRGAB image (generated pixels) for the arrangement of W pixels, R pixels, G pixels, A pixels, and B pixels are shown by dashed lines.

[0053] In the second demosaicing process, as shown in Figure 8, the region surrounding the four corners of the diamond shape of the W pixels in parallel horizontal and vertical directions corresponds to four generated pixels. The number of generated pixels in the W image is four times the number of pixels in the W pixels, and the resolution of the W image is four times the resolution of the W pixels in the pixel array. The resolution of the R image becomes 16 times the resolution of the R pixels. The resolution of the G image becomes 16 times the resolution of the G pixels. The resolution of the B image becomes 16 times the resolution of the B pixels.

[0054] As described above, the demosaicing unit 17 performs demosaicing and generates a WRGB image or WRGAB image with a resolution equal to or higher than the resolution of the W pixels of the image sensor 12. The resolution of the generated WRGB image or WRGAB image can be determined, for example, by a setting screen (not shown) or by setting information supplied to the control unit 22. The detailed configuration of the demosaicing unit 17 will be described later.

[0055] Returning to the explanation of Figure 1, the color processing unit 18 performs color correction and gamma correction on the WRGB or WRGAB image data output from the demosaicing unit 17. The color processing unit 18 outputs the RGB image data obtained through the color correction and gamma correction processes to the Y color difference conversion unit 19. The Y color difference conversion unit 19 converts the RGB image data output from the color processing unit 18 into Y color difference image data.

[0056] The codec unit 20 compresses the Y chromatic difference image data output from the Y chromatic difference conversion unit 19 using an encoder such as MPEG. The output unit 21 outputs the compressed image data from the codec unit 20. For example, the output unit 21 outputs the image data from the codec unit 20 to an external device via, for example, a wired or wireless network or various cables. The output unit 21 may also have a driver to drive recording media such as semiconductor memory, magnetic disks, or optical disks, and record the image data from the codec unit 20 onto the recording media. The control unit 22 controls each block constituting the imaging device 10 according to user operations, etc.

[0057] <2. Example of Demosaicing Configuration> <Example 1 configuration (WRGB array)> Figure 9 is a block diagram showing a first configuration example of the demosaicing section 17, specifically an example where the pixel array is a WRGB array.

[0058] In the following explanation, the signal value output from the W pixel of the image sensor 12 will be referred to as pixel value W. Similarly, the signal values ​​output from the R pixel, G pixel, and B pixel will be referred to as pixel value R, pixel value G, and pixel value B, respectively. Furthermore, in the following explanation, the basic process will be described when the resolution of the generated W image is equal to the resolution of the W pixel, and the case where the resolution of the W image is increased from the resolution of the W pixel will be explained supplementarily.

[0059] The demosaicing unit 17 includes a pixel of interest setting unit 31, an interpolation W' generation unit 32, a virtual color difference (R-W') generation unit 33, a virtual color difference (G-W') generation unit 34, and a virtual color difference (B-W') generation unit 35. The demosaicing unit 17 also includes a high-resolution W generation unit 36, a high-resolution (RW) generation unit 37, a high-resolution (GW) generation unit 38, a high-resolution (BW) generation unit 39, an adder 40, an adder 41, and an adder 42. The demosaicing unit 17 receives an image with the WRGB array shown in Figure 2 as an input image from the white balance unit 16.

[0060] The focus pixel setting unit 31 sets the pixels to be processed as focus pixels. For example, the focus pixel setting unit 31 scans each pixel constituting the input image or output image in a predetermined order, such as a raster scan, and sequentially sets them as focus pixels. As a result, each block of the demosaicing unit 17 performs image processing on the focus pixels that are being set.

[0061] The interpolation W' generation unit 32 is a first-color component interpolation generation unit that generates first-color component interpolated pixel values ​​that interpolate the first color component (W component). The interpolation W' generation unit 32 uses the W pixels of the input image to generate pixel values ​​W' equivalent to the W pixels (hereinafter also referred to as W' pixels as appropriate) for each pixel at the R pixel position, G pixel position, and B pixel position of the input image. The interpolation W' generation unit 32 outputs the generated pixel values ​​W' to the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, and the virtual color difference (B-W') generation unit 35.

[0062] The virtual color difference (R-W') generation unit 33 is a second-color component virtual color difference generation unit that generates a virtual color difference (R-W') of the second color component (R component) and generates a virtual color difference (R-W') image in which the generated virtual color difference (R-W') is stored as a pixel value at the R pixel position of the input image. The virtual color difference (R-W') generation unit 33 generates a virtual color difference (R-W') for each R pixel of the input image. The virtual color difference (R-W') is calculated as the difference between the pixel value of the R pixel and the pixel value of the W' pixel. Hereinafter, pixels that have a virtual color difference (R-W') will be referred to as virtual color difference (R-W') pixels. The virtual color difference (R-W') generation unit 33 outputs the generated virtual color difference (R-W') image to the high-resolution (RW) generation unit 37.

[0063] The virtual color difference (G-W') generation unit 33 is a third-color component virtual color difference generation unit that generates a virtual color difference (G-W') of the third color component (G component) and generates a virtual color difference (G-W') image in which the generated virtual color difference (G-W') is stored as a pixel value at the G pixel position of the input image. The virtual color difference (G-W') generation unit 34 generates a virtual color difference (G-W') for each G pixel of the input image. The virtual color difference (G-W') is calculated as the difference between the pixel value of the G pixel and the pixel value of the W' pixel. Hereinafter, pixels that have a virtual color difference (G-W') will be referred to as virtual color difference (G-W') pixels. The virtual color difference (G-W') generation unit 34 outputs the generated virtual color difference (G-W') image to the high-resolution (GW) generation unit 38.

[0064] The virtual color difference (B-W') generation unit 35 is a fourth-color component virtual color difference generation unit that generates a virtual color difference (B-W') of the fourth color component (B component) and generates a virtual color difference (B-W') image in which the generated virtual color difference (B-W') is stored as a pixel value at the B pixel position of the input image. The virtual color difference (B-W') generation unit 35 generates a virtual color difference (B-W') for each B pixel of the input image. The virtual color difference (B-W') is calculated as the difference between the pixel value of the B pixel and the pixel value of the W' pixel. Hereinafter, pixels that have a virtual color difference (B-W') will be referred to as virtual color difference (B-W') pixels. The virtual color difference (B-W') generation unit 35 outputs the generated virtual color difference (B-W') image to the high-resolution (BW) generation unit 39.

[0065] The high-resolution W generation unit 36, as shown in Figures 4 and 7, is a first-color component image generation unit that generates a W image in which each pixel has a pixel value for the first color component (W component), and has a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix. The high-resolution W generation unit 36 ​​generates a W image using the W pixels of the input image. As described above, the resolution of the generated W image can be the same as the input image, or it can be higher resolution than the input image. Below, we will mainly describe the case in which the W image has the same resolution as the input image. The high-resolution W generation unit 36 ​​outputs the generated W image to the subsequent color processing unit 18, as well as to adders 40, 41, and 42.

[0066] The high-resolution (RW) generation unit 37 generates a high-resolution (RW) image using the virtual color difference (R-W') image output from the virtual color difference (R-W') generation unit 33 and the W and R pixels of the input image. The high-resolution (RW) generation unit 37 is a second color difference component image generation unit that generates a color difference component image of the second color component (R component) and the first color component (W component).

[0067] The high-resolution (GW) generation unit 38 generates a high-resolution (GW) image using the virtual color difference (G-W') image output from the virtual color difference (G-W') generation unit 34 and the W and G pixels of the input image. The high-resolution (GW) generation unit 38 is a third-color component image generation unit that generates a color difference component image of the third color component (G component) and the first color component (W component).

[0068] The high-resolution (BW) generation unit 39 generates a high-resolution (BW) image using the virtual color difference (B-W') image output from the virtual color difference (B-W') generation unit 35 and the W pixels and B pixels of the input image. The high-resolution (BW) generation unit 39 is a fourth color difference component image generation unit that generates a color difference component image of the fourth color component (B component) and the first color component (W component).

[0069] Adder 40 (first adder) generates an R image by adding the W image output from the high-resolution W generation unit 36 ​​and the high-resolution (RW) image output from the high-resolution (RW) generation unit 37. Adder 41 (second adder) generates a G image by adding the W image output from the high-resolution W generation unit 36 ​​and the high-resolution (GW) image output from the high-resolution (GW) generation unit 38. Adder 42 (third adder) generates a B image by adding the W image output from the high-resolution W generation unit 36 ​​and the high-resolution (BW) image output from the high-resolution (BW) generation unit 39.

[0070] As described above, the demosaicing unit 17 in the first configuration example interpolates and generates pixel values ​​W' equivalent to W pixels (W' pixels) at the R pixel positions, G pixel positions, and B pixel positions of the input image. Subsequently, the demosaicing unit 17 generates virtual color differences (R-W'), (G-W'), and (B-W') using the pixel values ​​R, G, and B, and the interpolated pixel value W'. The demosaicing unit 17 generates high-resolution (RW) images, high-resolution (GW) images, and high-resolution (BW) images using these virtual color differences (R-W'), (G-W'), and (B-W'), along with the pixel values ​​W, R, G, and B. In addition, the demosaicing unit 17 generates a W image using only the W pixels of the input image. Furthermore, the demosaicing unit 17 generates R, G, and B images based on the W image and the high-resolution (RW), high-resolution (GW), and high-resolution (BW) images. The generated W, R, G, and B images are output to the subsequent color processing unit 18.

[0071] <Second configuration example (in the case of a WRGAB array)> Figure 10 is a block diagram showing a second configuration example of the demosaicing section 17, specifically an example where the pixel array is a WRGAB array.

[0072] In Figure 10, parts common to the first configuration example shown in Figure 9 are denoted by the same reference numerals, and explanations of those parts are omitted as appropriate.

[0073] As described above, the WRGAB array image sensor 12 has an additional A pixel. The signal value output from the A pixel of the image sensor 12 is referred to as pixel value A.

[0074] The demosaicing unit 17 includes a pixel of interest setting unit 31, an interpolation W' generation unit 32, a virtual color difference (R-W') generation unit 33, a virtual color difference (G-W') generation unit 34, and a virtual color difference (B-W') generation unit 35. The demosaicing unit 17 also includes a high-resolution W generation unit 36, a high-resolution (RW) generation unit 37, a high-resolution (GW) generation unit 38, a high-resolution (BW) generation unit 39, an adder 40, an adder 41, and an adder 42. Furthermore, the demosaicing unit 17 includes a virtual color difference (A-W') generation unit 43, a high-resolution (AW) generation unit 44, and an adder 45. Comparing the second configuration example with the first configuration example shown in Figure 9, the virtual color difference (A-W') generation unit 43, the high-resolution (AW) generation unit 44, and the adder 45, which are blocks corresponding to A pixels, have been added.

[0075] The virtual color difference (A-W') generation unit 43 is a fifth-color component virtual color difference generation unit that generates a virtual color difference (A-W') of the fifth color component (A component) and generates a virtual color difference (A-W') image in which the generated virtual color difference (A-W') is stored as a pixel value at the A pixel position of the input image. The virtual color difference (A-W') generation unit 43 generates a virtual color difference (A-W') for each A pixel of the input image. The virtual color difference (A-W') is calculated as the difference between the pixel value of the A pixel and the pixel value of the W' pixel. Hereinafter, pixels that have a virtual color difference (A-W') will be referred to as virtual color difference (A-W') pixels. The virtual color difference (A-W') generation unit 43 outputs the generated virtual color difference (A-W') image to the high-resolution (AW) generation unit 44.

[0076] The high-resolution (AW) generation unit 44 generates a high-resolution (AW) image using the virtual color difference (A-W') image output from the virtual color difference (A-W') generation unit 43 and the W pixels and A pixels of the input image. The high-resolution (AW) generation unit 44 is a fifth color difference component image generation unit that generates a color difference component image of the fifth color component (A component) and the first color component (W component).

[0077] The adder 45 (fourth adder) generates an A image by adding the W image output from the high-resolution W generation unit 36 ​​and the high-resolution (AW) image output from the high-resolution (AW) generation unit 44.

[0078] As described above, the demosaicing unit 17 in the second configuration example interpolates and generates pixel values ​​W' equivalent to W pixels (W' pixels) at the R pixel positions, G pixel positions, A pixel positions, and B pixel positions of the input image. Subsequently, the demosaicing unit 17 generates virtual color differences (R-W'), virtual color differences (G-W'), virtual color differences (A-W'), and virtual color differences (B-W') using the pixel values ​​R, G, A, and B, and the interpolated pixel value W'. The demosaicing unit 17 generates high-resolution (RW) images, high-resolution (GW) images, high-resolution (AW) images, and high-resolution (BW) images using these virtual color differences (R-W'), virtual color differences (G-W'), virtual color differences (A-W'), and virtual color differences (B-W'), along with the pixel values ​​W, R, G, A, and B. Furthermore, the demosaicing unit 17 generates a W image using only the W pixels of the input image. The demosaicing unit 17 then generates R, G, A, and B images based on this W image and the high-resolution (RW), high-resolution (GW), high-resolution (AW), and high-resolution (BW) images. The generated W, R, G, A, and B images are output to the subsequent color processing unit 18.

[0079] <3. Detailed Configuration Example of the Interpolation W' Generation Unit> Figure 11 is a block diagram showing a detailed configuration example of the interpolation W' generation unit 32.

[0080] The explanation for Figure 11 describes the case where the pixel array is a WRGB array, but the same applies to the WRGAB array. The same applies to the explanations from Figure 12 onward.

[0081] The interpolation W' generation unit 32 uses the W pixels of the input image to generate and output pixel values ​​W' corresponding to the W pixels (W' pixels) at the R pixel positions, G pixel positions, and B pixel positions of the input image.

[0082] The interpolation W' generation unit 32 includes a W' code tap selection unit 51, a W' filter tap selection unit 52, a W' code calculation unit 53, a W' coefficient storage unit 54, and a W' sum-of-products calculation unit 55. In the interpolation W' generation unit 32, each pixel at the R pixel position, G pixel position, and B pixel position of the input image for which the pixel value W' is to be generated is sequentially set as the target pixel by the target pixel setting unit 31.

[0083] The W' code tap selection unit 51 selects tap pixels (hereinafter referred to as W' code taps) that generate a code for generating the pixel value W' of the target pixel. The W' code taps are composed of, for example, a plurality of W pixels included in a predetermined peripheral region centered on the target pixel. The W' code taps selected by the W' code tap selection unit 51 are supplied to the W' code calculation unit 53.

[0084] <Example of W' code tap> FIG. 12 is a diagram showing an example of a W' code tap.

[0085] In the WRGB array of FIG. 12, the pixel located at the point where the cross-dashed line intersects is the target pixel for generating the pixel value W'. In the example of FIG. 12, the R pixel of the input image is the target pixel.

[0086] The W' code tap selection unit 51 selects, for example, four W pixels located around the target pixel as the W' code taps. In FIG. 12, the W pixels selected as the W' code taps are shown surrounded by a thick line. The pixel value W of the W pixels selected as the W' code taps is supplied to the W' code calculation unit 53. Note that the W' code taps are not limited to the selection example shown in FIG. 12 and may be composed of other multiple W pixels.

[0087] Returning to the description of FIG. 11, the W' code calculation unit 53 generates the W' code of the target pixel based on the W' code taps selected by the W' code tap selection unit 51. For example, the W' code calculation unit 53 extracts feature amounts using a method such as DR quantization that encodes the magnitude of the dynamic range of the selected W' code taps. Then, the W' code calculation unit 53 classifies the codes according to a predetermined rule using the information on the extracted feature amounts, and generates the W' code of the target pixel. The W' code calculation unit 53 supplies the generated W' code of the target pixel to the W' coefficient storage unit 54. Details of the W' code generation by the W' code calculation unit 53 will be described later.

[0088] In the W' coefficient storage unit 54, tap coefficients (third tap coefficients) for each W' code obtained by learning described later are stored in advance in association with the W' codes. The W' coefficient storage unit 54 reads out the tap coefficients stored at the address corresponding to the W' code supplied from the W' code calculation unit 53 from the already stored tap coefficients, and supplies them to the W' product-sum calculation unit 55.

[0089] The W' filter tap selection unit 52 selects a plurality of W pixels as filter taps (hereinafter referred to as W' filter taps) for generating the pixel value W' of the target pixel from the input image. The W' filter taps are composed of, for example, a plurality of W pixels included in a predetermined peripheral region centered on the target pixel. The W' filter tap selection unit 52 outputs the pixel value W of the W pixels selected as the W' filter taps to the W' product-sum calculation unit 55.

[0090] <Example of W' filter tap> FIG. 13 is a diagram showing an example of a W' filter tap.

[0091] In the WRGB array of FIG. 13, the pixel located at the intersection of the cross-dashed line is the target pixel for generating the pixel value W'. In the example of FIG. 13, as in FIG. 12, the R pixel of the input image is the target pixel.

[0092] The W' filter tap selection unit 52 selects, for example, 12 W pixels located in the surrounding area centered on the pixel of interest as the W' filter tap. In Figure 13, the W pixels selected as the W' filter tap are shown enclosed in a thick line. Note that the W' filter tap is not limited to the selection example shown in Figure 13, and may consist of other W pixels.

[0093] Returning to the explanation of Figure 11, the W' sum-of-products unit 55 generates (predicts) a pixel value W' that interpolates the W pixel for the pixel of interest by performing a predetermined filter operation using the tap coefficients supplied from the W' coefficient storage unit 54 and the pixel value W of the W' filter tap supplied from the W' filter tap selection unit 52. If we adopt a linear first-order operation as the predetermined filter operation, the pixel value W' of the pixel of interest can be obtained by the value of y in equation (1).

[0094]

number

[0095] Here, N is the number of pixels in the W' filter tap. n w is the pixel value W of each W pixel that constitutes the W' filter tap selected by the W' filter tap selection unit 52. n This is the tap coefficient supplied from the W' coefficient storage unit 54.

[0096] As described above, the W' sum-of-products unit 55 predicts the pixel value W' of the pixel of interest by performing the filter operation of equation (1).

[0097] The W' sum-of-products unit 55 outputs the generated pixel value W' to the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, and the virtual color difference (B-W') generation unit 35. If the pixel array is a WRGB array, the W' sum-of-products unit 55 also outputs the generated pixel value W' to the virtual color difference (A-W') generation unit 43.

[0098] The interpolation W'-generation unit 32 obtains a pixel value W' obtained by interpolating pixels other than W pixels as W' pixels corresponding to W pixels by sequentially selecting pixels at the R pixel position, G pixel position, and B pixel position of the input image as the target pixels.

[0099] In the above-described example, the interpolation W'-generation unit 32 is configured to calculate a W' code based on the W' code taps selected for the target pixels, obtain tap coefficients associated with the W' code, and perform a sum-of-products operation with the W' filter taps. When reducing the arithmetic processing, tap coefficients common to any input image can be adopted. In this case, the interpolation W'-generation unit 32 shown in FIG. 11 can be configured only by the W' filter tap selection unit 52 and the W' sum-of-products operation unit 55, with the W' code tap selection unit 51, the W' code calculation unit 53, and the W' coefficient storage unit 54 omitted. The W' filter tap selection unit 52 selects a plurality of W pixels around the target pixel as W' filter taps. The W' sum-of-products operation unit 55 performs a sum-of-products operation with the W' filter taps using tap coefficients that are calculated in advance by learning and stored in the internal memory. These tap coefficients are fixed coefficients that are commonly used for any input image. Thereby, the processing load of the interpolation W'-generation unit 32 can be reduced, and the pixel value W' of the W' pixel can be calculated at high speed.

[0100] <Detailed configuration example of W' code calculation unit> FIG. 14 is a block diagram showing a detailed configuration example of the W' code calculation unit 53 shown in FIG. 11.

[0101] The W' code calculation unit 53 includes a quantization calculation unit 56 and a conversion table storage unit 57. The W' code calculation unit 53 generates a W' code for the target pixel based on the W' code taps supplied from the W' code tap selection unit 51.

[0102] The quantization calculation unit 56 quantizes the pixel values W of a plurality of W pixels constituting the W' code taps and supplies a quantization code arranged in a predetermined order to the conversion table storage unit 57.

[0103] One method for quantizing pixel values ​​W is, for example, DR (Dynamic Range) quantization. In DR quantization, the pixel values ​​of each pixel constituting the W' code tap of the pixel of interest are quantized, and the DR quantized code obtained through quantization is used as the quantized code of the pixel of interest.

[0104] For example, in the case of N-bit DR quantization, the maximum value MAX and minimum value MIN of the pixel value of each pixel constituting the W' code tap are detected. Then, the local dynamic range DR (=MAX-MIN) of the set of pixels constituting the W' code tap is calculated. Based on this dynamic range DR, the pixel value of each pixel constituting the W' code tap is quantized to N bits. More specifically, the minimum value MIN is subtracted from the pixel value of each pixel constituting the W' code tap, and the resulting subtraction is DR / 2 N The values ​​are divided by (quantized). The pixel values ​​of each N-bit pixel that constitute the W' code tap obtained through the above process are arranged in a predetermined order, and the arranged bit sequence is output as the quantized code.

[0105] For example, when a code tap is DR quantized with 1 bit, the pixel value of each pixel constituting that code tap is divided by the average of the maximum value MAX and the minimum value MIN (integer arithmetic). As a result, the pixel value of each pixel is made 1 bit (binarized). Then, a bit sequence obtained by arranging these 1-bit pixel values ​​in a predetermined order is output as the quantized code. When code classification is performed using only DR quantization, for example, the DR quantized code becomes the W' code generated by the W' code calculation unit 53.

[0106] Figure 15 illustrates an example of 1-bit DR quantization. The horizontal axis represents the order (or position) of the W pixels that constitute the W' code tap. The vertical axis represents the pixel value W of the W pixels that constitute the W' code tap.

[0107] In 1-bit DR quantization, the simple dynamic range DR is calculated by subtracting the minimum pixel value Min from the maximum pixel value Max among the pixel values ​​W of the W pixels that make up the W' code tap. Then, using a threshold level that bisects the simple dynamic range DR, the pixel values ​​of each W pixel that make up the W' code tap are binarized and converted into a 1-bit W' code.

[0108] In the case of the W' code tap shown in Figure 12, when 1-bit DR quantization is performed on all four W' code taps, a 4-bit DR quantized code is obtained. The DR quantized code obtained by the quantization calculation unit 56 is supplied to the conversion table storage unit 57.

[0109] Furthermore, the W'code calculation unit 53 can also output, for example, the level distribution pattern of the pixel values ​​of the pixels constituting the W'code tap as the W'code itself. In this case, if the W'code tap consists of the pixel values ​​of M pixels and Q bits are assigned to the pixel value of each pixel, the number of W'code cases output from the W'code calculation unit 53 is (2 M ) Q This results in a vast number of possibilities, exponentially proportional to the number of bits Q in the pixel value.

[0110] Therefore, it is preferable for the W' code calculation unit 53 to perform code classification by compressing the amount of information in the W' code tap by the aforementioned DR quantization or vector quantization or the like.

[0111] The conversion table storage unit 57 stores a conversion table. This conversion table optimizes the DR quantization code (input code) obtained by the quantization calculation unit 56 and converts it into a code (output code) that the W' coefficient storage unit 54 can handle. Therefore, the DR quantization code obtained by the quantization calculation unit 56 is converted into a code for the W' coefficient storage unit 54 by the conversion table pre-stored in the conversion table storage unit 57. The converted code is then supplied to the W' coefficient storage unit 54 as the W' code.

[0112] If the storage capacity of the W' coefficient storage unit 54 is sufficiently large, the conversion table storage unit 57 can be omitted. In that case, the DR quantization code obtained from the quantization calculation unit 56 is supplied directly to the W' coefficient storage unit 54 as the W' code.

[0113] Furthermore, the conversion table stored in the conversion table storage unit 57 can be updated as needed to optimize the output codes. This allows the conversion table to, for example, replace the output code corresponding to a particular input code with a representative code if that input code occurs infrequently. Additionally, if different output codes are associated with similar coefficients, the conversion table may treat them as the same code.

[0114] The interpolation W' generation unit 32, having the above configuration, generates pixel values ​​W' for W pixel equivalents (W' pixels) at the R pixel positions, G pixel positions, and B pixel positions of the input image by interpolation using the W pixels of the input image. The interpolation W' generation unit 32 outputs the pixel values ​​W' of the generated W' pixels to the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, and the virtual color difference (B-W') generation unit 35. If the pixel array is a WRGAB array, the pixel values ​​W' of the generated W' pixels are also output to the virtual color difference (A-W') generation unit 43.

[0115] <4. Detailed Configuration Example of High-Resolution W Generation Unit> Figure 16 is a block diagram showing a detailed configuration example of the high-resolution W generation unit 36.

[0116] The high-resolution W generation unit 36 ​​uses only the W pixels of the input image to generate a W image with the same resolution as the input image, or a higher resolution than the input image. The W pixels generated by the high-resolution W generation unit 36 ​​are distinguished from the W pixels of the input image and are called high-resolution W pixels, and the pixel value W of the high-resolution W pixels is called the high-resolution pixel value W. For simplicity, the following explanation will describe the case in which a W image with the same resolution as the input image is generated. In the high-resolution W generation unit 36, each pixel of the W image to be generated is set as a pixel of interest by the pixel of interest setting unit 31 and processed.

[0117] The high-resolution W generation unit 36 includes a W code tap selection unit 61, a W filter tap selection unit 62, a W code calculation unit 63, a W coefficient storage unit 64, and a W sum-of-products calculation unit 65.

[0118] The W code tap selection unit 61 selects tap pixels (hereinafter referred to as W code taps) that generate codes for generating the high-resolution pixel value W of the target pixel. The W code taps are composed of, for example, a plurality of W pixels included in a predetermined peripheral region centered on the pixel of the input image corresponding to the target pixel. The W' code taps selected by the W' code tap selection unit 51 are supplied to the W code calculation unit 63.

[0119] <Example of W code tap> FIG. 17 is a diagram showing an example of a W code tap.

[0120] In the WRGB array of FIG. 17, the dotted rectangular pixel centered on the point where the cross-dashed line intersects is the target pixel for generating the high-resolution pixel value W. Therefore, for the diamond-shaped pixels in the WRGB array, the generated high-resolution W pixels are pixels that are, so to speak, area-corrected and are composed of a region that horizontally and vertically surrounds the four corners of the diamond. The high-resolution W pixels are also generated at pixel positions that are not the W pixels of the input image.

[0121] The W code tap selection unit 61 selects, for example, a plurality of W pixels included in a specified region based on the W pixel closest to the pixel position of the input image corresponding to the target pixel as the W code taps. In FIG. 17, the W code taps selected by the W code tap selection unit 61 are surrounded by a thick line, and nine 3x3 W pixels are selected as the W code taps. The pixel value W of the W pixels selected as the W code taps is supplied to the W code calculation unit 63. Note that the W code taps are not limited to the selection example shown in FIG. 17 and may be composed of other multiple W pixels.

[0122] The W-code calculation unit 63 generates the W-code (first code) of the target pixel based on the W-code taps selected by the W-code tap selection unit 61. The method for generating the W-code can adopt the same method as that of the above-described W'-code calculation unit 53, such as DR quantization. The W-code calculation unit 63 supplies the W-code of the target pixel to the W-coefficient storage unit 64.

[0123] In the W-coefficient storage unit 64, the tap coefficients (first tap coefficients) for each W-code obtained by the learning described later are stored in advance in association with the W-codes. The W-coefficient storage unit 64 reads out the tap coefficients stored at the address corresponding to the W-code supplied from the W-code calculation unit 63 from the already stored tap coefficients, and supplies them to the W-product sum calculation unit 65.

[0124] The W-filter tap selection unit 62 selects a plurality of W-pixels for generating the high-resolution pixel value W of the target pixel as filter taps (hereinafter referred to as W-filter taps) from the input image. The W-filter taps are composed of, for example, a plurality of W-pixels included in a predetermined peripheral region centered on the pixel of the input image corresponding to the target pixel. The W-filter tap selection unit 62 outputs the pixel value W of the W-pixels selected as the W-filter taps to the W-product sum calculation unit 65.

[0125] <Example of W-filter tap> FIG. 18 is a diagram showing an example of a W-filter tap.

[0126] In the WRGB array of FIG. 18, the dotted rectangular pixel centered on the point where the cross-dashed line intersects is the target pixel for generating the high-resolution pixel value W. Therefore, for the diamond-shaped pixels in the WRGB array, the generated high-resolution W-pixels are pixels that are, so to speak, area-corrected and are composed of a region that horizontally and vertically surrounds the four corners of the diamond.

[0127] The W filter tap selection unit 62 selects, for example, multiple W pixels included in a specified region based on the W pixel closest to the pixel position of the input image corresponding to the pixel of interest, as a W filter tap. In Figure 18, the W filter tap selected by the W filter tap selection unit 62 is enclosed by a thick line, and nine 3x3 W pixels are selected as a W filter tap. The pixel value W of the W pixels selected as a W filter tap is supplied to the W sum-of-products unit 65. Note that the W filter tap is not limited to the selection example shown in Figure 18, and may consist of other multiple W pixels.

[0128] The W multiply-accumulate unit 65 generates (predicts) the high-resolution pixel value W of the high-resolution W pixel of interest by performing a predetermined filter operation using the tap coefficients supplied from the W coefficient storage unit 64 and the pixel value W of the W filter tap supplied from the W filter tap selection unit 62. Specifically, the W multiply-accumulate unit 65 predicts the high-resolution pixel value W of the high-resolution W pixel of interest using the linear first-order operation of equation (1) described above. In this case, N in equation (1) is the number of pixels of the W filter tap. n w is the pixel value W of each W pixel that constitutes the W filter tap selected by the W filter tap selection unit 62. n This is the tap coefficient read from the W coefficient storage unit 64.

[0129] As described above, the W multiply-accumulate unit 65 predicts the high-resolution pixel value W of the target pixel set for the W image to be generated by performing the filter operation of equation (1). The W image is generated by sequentially setting each pixel of the W image to be generated as the target pixel. The generated W image is supplied to adder 40, adder 41, and adder 42, as well as to the subsequent color processing unit 18. If the pixel array is a WRGAB array, the generated W image is also supplied to adder 45.

[0130] To reduce computational processing, a common tap coefficient can be used for any input image, similar to the interpolation W' generation unit 32 described above. In this case, the high-resolution W generation unit 36 ​​in Figure 16 omits the W code tap selection unit 61, the W code calculation unit 63, and the W coefficient storage unit 64, and consists only of the W filter tap selection unit 62 and the W sum-of-products calculation unit 65. The W filter tap selection unit 62 selects multiple W pixels around the pixel of interest as W filter taps. The W' sum-of-products calculation unit 55 uses tap coefficients that have been pre-calculated by learning and stored in internal memory to perform a sum-of-products calculation with the pixel value W of the W filter tap. This reduces the processing load of the high-resolution W generation unit 36, and the high-resolution pixel value W is calculated at high speed.

[0131] <5. Detailed Configuration Example of High-Resolution (RW) Generation Unit>

[0132] Figure 19 is a block diagram showing a detailed configuration example of the high-resolution (RW) generation unit 37.

[0133] The high-resolution (RW) generation unit 37 generates a high-resolution (RW) image using the virtual color difference (R-W') output from the virtual color difference (R-W') generation unit 33 and the W and R pixels of the input image. The resolution of this high-resolution (RW) image is the same as the resolution of the W image generated by the high-resolution W generation unit 36. Hereinafter, the pixels of the high-resolution (RW) image generated by setting the pixels of interest as appropriate will be called high-resolution (RW) pixels, and their pixel values ​​will be called high-resolution pixel values ​​(RW). The high-resolution pixel value (RW) is the pixel value of the color difference component (second color difference component) of the first color component (W component) and the second color component (R component).

[0134] The high-resolution (RW) generation unit 37 includes a color change amount code calculation unit 70, an (RW) code tap selection unit 71, an (RW) filter tap selection unit 72, an (RW) code calculation unit 73, an (RW) coefficient storage unit 74, and an (RW) sum-of-accumulate calculation unit 75. In the high-resolution (RW) generation unit 37, each pixel of the high-resolution (RW) image to be generated is set as a pixel of interest and processed by the pixel of interest setting unit 31.

[0135] The color change code calculation unit 70 calculates the dynamic range of the surrounding region of the pixel of interest for each of the W pixels and R pixels of the input image. Here, the dynamic range of the W pixels in the surrounding region of the pixel of interest is called the W dynamic range, and the dynamic range of the R pixels in the surrounding region of the pixel of interest is called the R dynamic range. The color change code calculation unit 70 determines the ratio of the W dynamic range to the R dynamic range (W dynamic range / R dynamic range) as the color change code of the pixel of interest and outputs it to the (RW) code calculation unit 73. Note that the color change code is not limited to the ratio of the dynamic ranges and may be determined using other feature quantities. For example, the color change code may be determined using activity, which is composed of the difference value between the pixel of interest and the pixels in its surrounding region.

[0136] <Example of dynamic range calculation range> Figure 20 shows an example of the dynamic range calculation range set to calculate the dynamic range of a pixel of interest.

[0137] In the WRGB array shown in Figure 20, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for generating the high-resolution (RW) image. Therefore, for the diamond-shaped pixels in the WRGB array, the generated high-resolution (RW) pixel is an area-corrected pixel, just like the high-resolution W pixel.

[0138] In the example shown in Figure 20, the dynamic range calculation area is set to include 25 W pixels (5x5 grid) and 9 R pixels (3x3 grid) centered on the pixel of interest. Note that the dynamic range calculation area is not limited to the example shown in Figure 20 and may consist of other areas.

[0139] The (RW) code tap selection unit 71 in Figure 19 selects tap pixels (hereinafter referred to as (RW) code taps) for generating codes to generate high-resolution pixel values ​​(RW) from the virtual color difference (R-W') image generated by the virtual color difference (R-W') generation unit 33. An (RW) code tap consists of, for example, multiple virtual color difference (R-W') pixels included in a predetermined peripheral region centered on the virtual color difference (R-W') pixel in the virtual color difference (R-W') image corresponding to the pixel of interest. The (RW) code taps selected by the (RW) code tap selection unit 71 are supplied to the (RW) code calculation unit 73.

[0140] <(RW) Code Tap Example> Figure 21 shows an example of a (RW) cord tap.

[0141] In the WRGB array in Figure 21, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which high-resolution pixel values ​​(RW) are generated. Therefore, the generated high-resolution pixel values ​​(RW) are area-corrected pixels for the diamond-shaped pixels in the WRGB array. Furthermore, since the high-resolution (RW) image is an image with the same resolution as the W image generated by the high-resolution W generation unit 36, it is also generated at pixel positions that are not virtual color difference (R-W') pixels. Consequently, the generated high-resolution pixel values ​​(RW) are also generated at positions that are phase-shifted diagonally from the input virtual color difference (R-W') pixels.

[0142] The (RW) code tap selection unit 71 selects, for example, multiple virtual color difference (R-W') pixels included in a specified area based on the virtual color difference (R-W') pixel closest to the input image pixel position corresponding to the pixel of interest as the (RW) code tap. In Figure 21, the (RW) code tap selected by the (RW) code tap selection unit 71 is enclosed in a thick line, indicating that the virtual color difference (R-W') pixel closest to the input image pixel position corresponding to the pixel of interest, and a total of four virtual color difference (R-W') pixels above, below, to the left and to the right of it, have been selected as the (RW) code tap. The virtual color difference (R-W') pixels selected as the (RW) code tap are supplied to the (RW) code calculation unit 73. Note that the (RW) code tap is not limited to the configuration example shown in Figure 21, and may be composed of other multiple virtual color difference (R-W') pixels.

[0143] The (RW) code calculation unit 73 generates the (RW) code (second code) of the pixel of interest using the (RW) code tap selected by the (RW) code tap selection unit 71. The method for generating the (RW) code can be the same as in the case of the W' code calculation unit 53 described above, for example, DR quantization. The (RW) code calculation unit 73 uses the (RW) code tap selected by the (RW) code tap selection unit 71 to generate the code by DR quantization or the like, and combines it with the color change amount code supplied from the color change amount code calculation unit 70 to obtain the (RW) code of the pixel of interest. For example, the (RW) code calculation unit 73 supplies the code obtained by bit-concatenating the (RW) code as the upper bit and the color change amount code as the lower bit to the (RW) coefficient storage unit 74 as the (RW) code of the pixel of interest.

[0144] The (RW) coefficient storage unit 74 stores in advance the tap coefficients (second tap coefficients) for each (RW) code, which have been determined by learning as described later, and associate them with the (RW) codes. The (RW) coefficient storage unit 74 reads out the tap coefficients stored at the address corresponding to the (RW) code supplied by the (RW) code calculation unit 73 from the tap coefficients already stored and supplies them to the (RW) multiply-accumulate unit 75.

[0145] The (RW) filter tap selection unit 72 selects a plurality of virtual color difference (R-W') pixels from the virtual color difference (R-W') image as filter taps (hereinafter referred to as (RW) filter taps) in order to generate the high-resolution pixel value (RW) of the pixel of interest. The (RW) filter tap selection unit 72 supplies the pixel values ​​of the virtual color difference (R-W') pixels selected as (RW) filter taps to the (RW) sum-of-products calculation unit 75.

[0146] <(RW) Filter Tap Example> Figure 22 shows an example of a (RW) filter tap.

[0147] In the WRGB array shown in Figure 22, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for generating the high-resolution (RW) image. Therefore, the generated high-resolution (RW) pixel may be a pixel that has undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixel in the WRGB array.

[0148] The (RW) filter tap selection unit 72 selects, for example, multiple virtual color difference (R-W') pixels included in a specified region based on the virtual color difference (R-W') pixel closest to the pixel position of the input image corresponding to the pixel of interest, as the (RW) filter tap. In Figure 22, the (RW) code tap selected by the (RW) code tap selection unit 71 is enclosed in a thick line, and the same four virtual color difference (R-W') pixels as the (RW) code tap are selected as the (RW) filter tap. The virtual color difference (R-W') pixels selected as the (RW) filter tap are supplied to the (RW) code calculation unit 73. Note that the (RW) filter tap is not limited to the configuration example shown in Figure 22, and may be composed of other multiple virtual color difference (R-W') pixels. In the example in Figure 22, the same pixels as the (RW) code tap are shown as the (RW) filter tap, but the (RW) code tap and the (RW) filter tap may be different.

[0149] (R-W) product-sum operation unit 75 generates (predicts) the high-resolution pixel value (R-W) of the high-resolution (R-W) pixel, which is the pixel of interest, through a predetermined filtering operation using the tap coefficients supplied from the (R-W) coefficient storage unit 74 and the (R-W) filter taps supplied from the (R-W) filter tap selection unit 72. Specifically, (R-W) product-sum operation unit 75 predicts the high-resolution pixel value (R-W) of the high-resolution (R-W) pixel, which is the pixel of interest, using the linear first-order operation of equation (1) described above. In this case, N in equation (1) is the number of pixels of the (R-W) filter tap. x n is the pixel value of each virtual color difference (R-W’) pixel constituting the (R-W) filter tap selected by the (R-W) filter tap selection unit 72. w n is the tap coefficient read from the (R-W) coefficient storage unit 74.

[0150] As described above, (R-W) product-sum operation unit 75 predicts the high-resolution pixel value (R-W) of the pixel of interest set for the generated high-resolution (R-W) image by performing the filtering operation of equation (1). By sequentially setting each pixel of the generated high-resolution (R-W) image as the pixel of interest, a high-resolution (R-W) image is generated. The generated high-resolution (R-W) image is supplied to adder 40.

[0151] When reducing the arithmetic processing, as in the case of the interpolation W’ generation unit 32 described above, tap coefficients common to any input image can be adopted. In this case, the high-resolution (R-W) generation unit 37 in FIG. 19 omits the color change amount code arithmetic unit 70, the (R-W) code tap selection unit 71, the (R-W) code arithmetic unit 73, and the (R-W) coefficient storage unit 74, and is composed only of the (R-W) filter tap selection unit 72 and the (R-W) product-sum operation unit 75. The (R-W) filter tap selection unit 72 selects a plurality of virtual color difference (R-W’) pixels around the pixel of interest as the (R-W) filter taps. The (R-W) product-sum operation unit 75 performs a product-sum operation with the (R-W) filter taps using the tap coefficients calculated in advance by learning and stored in the internal memory. As a result, the processing load of the high-resolution (R-W) generation unit 37 is reduced, and the high-resolution pixel value (R-W) is calculated at high speed.

[0152] <6. Detailed Configuration Example of the High-Resolution (GW) Generation Unit> Although not shown in the diagram, the high-resolution (GW) generation unit 38 has the same configuration as the high-resolution (RW) generation unit 37 shown in Figure 19, and performs the same processing as the high-resolution (RW) generation unit 37, using G pixels instead of R pixels.

[0153] The high-resolution (GW) generation unit 38 generates a high-resolution (GW) image using the virtual color difference (G-W') image output from the virtual color difference (G-W') generation unit 34 and the W pixels and G pixels of the input image. The pixel values ​​of the high-resolution (GW) image are converted to high-resolution pixel values ​​(RW). The high-resolution pixel values ​​(RW) are the pixel values ​​of the color difference component (third color difference component) of the first color component (W component) and the third color component (G component).

[0154] In the color change amount code calculation unit 70 of the high-resolution (GW) generation unit 38, the ratio of the W dynamic range to the G dynamic range (W dynamic range / G dynamic range) is determined as the color change amount code of the pixel of interest.

[0155] <(GW) Code Tap Example> Figure 23 shows an example of a (GW) code tap selected by the high-resolution (GW) generation unit 38.

[0156] In the WRGB array shown in Figure 23, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which the high-resolution pixel value (GW) is generated. Therefore, the generated high-resolution pixel value (GW) can be a pixel that has undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGB array.

[0157] In Figure 23, the (GW) code tap is enclosed by a thick line, and eight virtual color difference (G-W') pixels located near the pixel position of the input image corresponding to the pixel of interest are selected as the (GW) code tap. Note that the (GW) code tap is not limited to the configuration example shown in Figure 23, and may be composed of multiple other virtual color difference (G-W') pixels.

[0158] <(GW) Filter Tap Example> Figure 24 shows an example of a (GW) filter tap selected by the high-resolution (GW) generation unit 38.

[0159] In the WRGB array shown in Figure 24, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which the high-resolution pixel value (GW) is generated. Therefore, the generated high-resolution pixel value (GW) can be a pixel that has undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGB array.

[0160] In Figure 24, the (GW) filter tap is enclosed by a thick line, and eight virtual color difference (G-W') pixels located near the pixel position of the input image corresponding to the pixel of interest are selected as the (GW) filter tap. Note that the (GW) filter tap is not limited to the configuration example shown in Figure 24, and may be composed of multiple other virtual color difference (G-W') pixels. In the example in Figure 24, the same pixels as the (GW) code tap are selected as the (GW) filter tap, but the (GW) code tap and the (GW) filter tap may be different.

[0161] The high-resolution (GW) generation unit 38 predicts the high-resolution pixel value (GW) of a designated pixel of interest for the high-resolution (GW) image to be generated by performing the filter operation of equation (1). A high-resolution (GW) image is generated by sequentially setting each pixel of the high-resolution (GW) image to be generated as a designated pixel. The generated high-resolution (GW) image is supplied to the adder 41. The configuration for reducing the processing load of the high-resolution (GW) generation unit 38 is the same as that of the high-resolution (RW) generation unit 37 described above.

[0162] <7. Detailed Configuration Example of the High-Resolution (BW) Generation Unit> Although not shown in the diagram, the high-resolution (BW) generation unit 39 has the same configuration as the high-resolution (RW) generation unit 37 shown in Figure 19, and performs the same processing as the high-resolution (RW) generation unit 37, using B pixels instead of R pixels.

[0163] The high-resolution (BW) generation unit 39 generates a high-resolution (BW) image using the virtual color difference (B-W') image output from the virtual color difference (B-W') generation unit 35 and the W pixels and B pixels of the input image. The pixel values ​​of the high-resolution (BW) image are set to high-resolution pixel values ​​(BW). The high-resolution pixel values ​​(BW) are the pixel values ​​of the color difference component (fourth color difference component) of the first color component (W component) and the fourth color component (B component).

[0164] In the color change amount code calculation unit 70 of the high-resolution (BW) generation unit 39, the ratio of the W dynamic range to the B dynamic range (W dynamic range / B dynamic range) is determined as the color change amount code of the pixel of interest.

[0165] <(BW) Code Tap Example> Figure 25 shows an example of a (BW) code tap selected by the high-resolution (BW) generation unit 39.

[0166] In the WRGB array shown in Figure 25, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which the high-resolution pixel value (BW) is generated. Therefore, the generated high-resolution pixel value (BW) can be a pixel that has undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGB array.

[0167] In Figure 25, the (BW) code tap is enclosed by a thick line, and the (BW) code tap is selected from the four virtual color difference (B-W') pixels closest to the input image pixel position corresponding to the pixel of interest, and the pixels above, below, left, and right of it. Note that the (BW) code tap is not limited to the configuration example shown in Figure 25, and may be composed of other multiple virtual color difference (B-W') pixels.

[0168] <(BW) Filter Tap Example> Figure 26 shows an example of a (BW) filter tap selected by the high-resolution (BW) generation unit 39.

[0169] In the WRGB array shown in Figure 26, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which the high-resolution pixel value (BW) is generated. Therefore, the generated high-resolution pixel value (BW) can be a pixel that has undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGB array.

[0170] In Figure 26, the (BW) filter tap is enclosed by a thick line, and the same four virtual color difference (B-W') pixels as the (BW) code tap are selected as the (BW) filter tap. Note that the (BW) filter tap is not limited to the configuration example shown in Figure 26, and may be composed of other multiple virtual color difference (B-W') pixels. In the example in Figure 26, the same pixels as the (BW) code tap are selected as the (BW) filter tap, but the (BW) code tap and the (BW) filter tap may be different.

[0171] The high-resolution (BW) generation unit 39 predicts the high-resolution pixel value (BW) of a designated pixel of interest for the high-resolution (BW) image to be generated by performing the filter operation of equation (1). A high-resolution (BW) image is generated by sequentially setting each pixel of the high-resolution (BW) image to be generated as a designated pixel. The generated high-resolution (BW) image is supplied to the adder 42. The configuration for reducing the processing load of the high-resolution (BW) generation unit 39 is the same as that of the high-resolution (RW) generation unit 37 described above.

[0172] <8. Detailed Configuration Example of the High Resolution (AW) Generation Unit> Although not shown in the diagram, the high-resolution (AW) generation unit 44 has the same configuration as the high-resolution (RW) generation unit 37 shown in Figure 19, and performs the same processing as the high-resolution (RW) generation unit 37, using A pixels instead of R pixels.

[0173] The high-resolution (AW) generation unit 44 generates a high-resolution (AW) image using the virtual color difference (A-W') image output from the virtual color difference (A-W') generation unit 43 and the W pixels and A pixels of the input image. The pixel values ​​of the high-resolution (AW) image are set to high-resolution pixel values ​​(AW). The high-resolution pixel values ​​(AW) are the pixel values ​​of the color difference component (fifth color difference component) of the first color component (W component) and the fifth color component (A component).

[0174] In the color change amount code calculation unit 70 of the high-resolution (AW) generation unit 44, the ratio of the W dynamic range to the A dynamic range (W dynamic range / A dynamic range) is determined as the color change amount code of the pixel of interest.

[0175] <(AW) Code Tap Example> Figure 27 shows an example of an (AW) code tap selected by the high-resolution (AW) generation unit 44.

[0176] In the WRGAB array shown in Figure 27, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which high-resolution pixel values ​​(AW) are generated. Therefore, the generated high-resolution pixel values ​​(AW) can be pixels that have undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGAB array.

[0177] In Figure 27, the (AW) code tap is enclosed by a thick line, and the (AW) code tap is selected from the four virtual color difference (A-W') pixels closest to the input image pixel position corresponding to the pixel of interest, and the pixels above, below, left, and right of it. Note that the (AW) code tap is not limited to the configuration example shown in Figure 27, and may be composed of other multiple virtual color difference (A-W') pixels.

[0178] <(AW) Filter Tap Example> Figure 28 shows an example of an (AW) filter tap selected by the high-resolution (AW) generation unit 44.

[0179] In the WRGAB array shown in Figure 28, the dotted rectangular pixel centered at the point where the cross-shaped dashed lines intersect is the pixel of interest for which high-resolution pixel values ​​(AW) are generated. Therefore, the generated high-resolution pixel values ​​(AW) can be pixels that have undergone area correction and phase shift in the diagonal direction compared to the diamond-shaped pixels in the WRGAB array.

[0180] In Figure 28, the (AW) filter tap is enclosed by a thick line, and the same four virtual color difference (A-W') pixels as the (AW) code tap are selected as the (AW) filter tap. Note that the (AW) filter tap is not limited to the configuration example shown in Figure 28, and may be composed of other multiple virtual color difference (A-W') pixels. In the example in Figure 28, the same pixels as the (AW) code tap are selected as the (AW) filter tap, but the (AW) code tap and the (AW) filter tap may be different.

[0181] The high-resolution (AW) generation unit 44 predicts the high-resolution pixel value (AW) of a designated pixel of interest for the high-resolution (AW) image to be generated by performing the filter operation of equation (1). A high-resolution (AW) image is generated by sequentially setting each pixel of the high-resolution (AW) image to be generated as a designated pixel. The generated high-resolution (AW) image is supplied to the adder 45. The configuration for reducing the processing load of the high-resolution (AW) generation unit 44 is the same as that of the high-resolution (RW) generation unit 37 described above.

[0182] <9. Demosaicing Flowchart> Next, the demosaicing process by the demosaicing unit 17 will be explained with reference to the flowchart in Figure 29. Figure 29 describes the demosaicing process by the demosaicing unit 17 in a second configuration example where the pixel array is a WRGAB array. This process is started, for example, when WRGAB array image data is supplied from the white balance unit 16.

[0183] In steps S11 to S17, the interpolation W' generation unit 32 performs the W' pixel generation process. The W' pixel generation process generates the pixel value W' of the W pixel equivalent (W' pixel) at the R pixel position, G pixel position, and B pixel position of the input image, using the W pixel of the input image.

[0184] First, in step S11, the focus pixel setting unit 31 sets a predetermined pixel at the R pixel position, G pixel position, and B pixel position of the input image that will be used to generate the pixel value W' as a focus pixel. For example, the focus pixel setting unit 31 sets the first pixel in the raster scan order among the pixels at the R pixel position, G pixel position, and B pixel position of the input image that have not yet been set as a focus pixel as a focus pixel.

[0185] In step S12, the W' code tap selection unit 51 selects a W' code tap to generate the pixel value W' of the pixel of interest. For example, as explained with reference to Figure 12, multiple W pixels included in a predetermined surrounding area centered on the pixel of interest are selected as a W' code tap and supplied to the W' code calculation unit 53.

[0186] In step S13, the W' code calculation unit 53 extracts feature quantities based on the W' code taps supplied from the W' code tap selection unit 51 and generates the W' code of the pixel of interest. For example, the W' code can be generated using DR quantization, as explained with reference to Figure 15.

[0187] In step S14, the W' filter tap selection unit 52 selects a W' filter tap for the pixel of interest. A W' filter tap consists of multiple W pixels included in a predetermined surrounding area centered on the pixel of interest, as explained with reference to Figure 13, for example. The selected W' filter tap is supplied to the W' sum-of-products unit 55.

[0188] In step S15, the W' coefficient storage unit 54 reads the tap coefficient stored at the address corresponding to the W' code supplied by the W' code calculation unit 53 and supplies it to the W' sum-of-products calculation unit 55.

[0189] In step S16, the W' sum-accumulate unit 55 generates a pixel value W' that interpolates the W pixel for the pixel of interest by performing a sum-accumulate operation. Specifically, the W' sum-accumulate unit 55 performs the linear first-order operation of equation (1) using the tap coefficient supplied from the W' coefficient storage unit 54 and the pixel value W of the W' filter tap supplied from the W' filter tap selection unit 52. The generated pixel value W' is supplied to the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, the virtual color difference (B-W') generation unit 35, and the virtual color difference (A-W') generation unit 43.

[0190] In step S17, the focus pixel setting unit 31 determines whether there are any W' pixels that have not yet been set as focus pixels, that is, whether there are any pixels at the R, G, and B pixel positions of the input image that have not yet been set as focus pixels. If it is determined in step S17 that there are W' pixels that have not yet been set as focus pixels, the process returns to step S11. As a result, the pixels following the R, G, and B pixel positions of the input image that have not yet been set as focus pixels in the raster scan order are set as focus pixels. On the other hand, if it is determined in step S17 that there are no W' pixels that have not yet been set as focus pixels, that is, all pixels at the R, G, and B pixel positions of the input image have been set as focus pixels, the process proceeds to step S18.

[0191] In step S18, a virtual color difference (R-W') image, a virtual color difference (G-W') image, a virtual color difference (B-W') image, and a virtual color difference (A-W') image are generated. More specifically, the virtual color difference (R-W') generation unit 33 generates a virtual color difference (R-W') at the R pixel position of the input image using the pixel value R of the R pixel of the input image and the pixel value W' generated for the position of that R pixel. Then, the virtual color difference (R-W') image, in which the virtual color difference (R-W') is stored at each R pixel position of the input image, is supplied to the high-resolution (RW) generation unit 37. The virtual color difference (G-W') generation unit 34 generates a virtual color difference (G-W') at the G pixel position of the input image using the pixel value G of the G pixel of the input image and the pixel value W' generated for the position of that G pixel. Then, a virtual color difference (G-W') image, in which virtual color differences (G-W') are stored at each G pixel position of the input image, is supplied to the high-resolution (GW) generation unit 38. The virtual color difference (B-W') generation unit 35 generates virtual color differences (B-W') at the B pixel positions of the input image using the pixel value B of the B pixels of the input image and the pixel value W' generated for the position of the B pixels. Then, a virtual color difference (B-W') image, in which virtual color differences (B-W') are stored at each B pixel position of the input image, is supplied to the high-resolution (BW) generation unit 39. The virtual color difference (A-W') generation unit 43 generates virtual color differences (A-W') at the A pixel positions of the input image using the pixel value A of the A pixels of the input image and the pixel value W' generated for the position of the A pixels. Then, a virtual color difference (A-W') image, in which virtual color differences (A-W') are stored at each A pixel position of the input image, is supplied to the high-resolution (AW) generation unit 44.

[0192] In the next step S19, the high-resolution W generation unit 36 ​​generates a W image using only the W pixels of the input image. The resolution of the generated W image can be set to the same resolution as the input image or to a higher resolution, depending on the settings screen or other setting information. The W image generation process is basically the same as the process of generating the pixel value W' in steps S11 to S17 described above, so a detailed explanation is omitted. In the process of generating the pixel value W' of the W' pixels, the pixels at the R, G, and B pixel positions of the input image were used as the pixels of interest, but in the generation of the W image, each pixel of the output image W image is used as the pixels of interest. Also, the W' code tap and W' filter tap described above are changed to W code tap and W filter tap. The generated W image is supplied to adder 40, adder 41, adder 42, and adder 45, as well as to the subsequent color processing unit 18.

[0193] In the next steps, S20 to S27, the processes for generating high-resolution (RW) images, high-resolution (GW) images, high-resolution (BW) images, and high-resolution (AW) images are executed in parallel in the high-resolution (RW) generation unit 37, high-resolution (GW) generation unit 38, high-resolution (BW) generation unit 39, and high-resolution (AW) generation unit 44, respectively. In the next steps, S20 to S27, the processing of the high-resolution (RW) generation unit 37 will be mainly described, and the processing of the high-resolution (GW) generation unit 38, high-resolution (BW) generation unit 39, and high-resolution (AW) generation unit 44 will be supplemented.

[0194] First, in step S20, the focus pixel setting unit 31 sets a predetermined pixel of the high-resolution (RW) image, high-resolution (GW) image, high-resolution (BW) image, and high-resolution (AW) image to be generated as the output image as the focus pixel. For example, the focus pixel setting unit 31 sets a pixel of the output image that has not yet been designated as a focus pixel in the raster scan order as the focus pixel.

[0195] In step S21, the color change code calculation unit 70 calculates and determines the color change code of the pixel of interest and outputs it to the (RW) code calculation unit 73. For example, the color change code calculation unit 70 uses the W and R pixels in the surrounding area of ​​the pixel of interest to calculate the W dynamic range and R dynamic range, and uses the ratio of the W dynamic range to the R dynamic range as the color change code of the pixel of interest. In the high-resolution (GW) generation unit 38, the ratio of the W dynamic range to the G dynamic range is determined as the color change code of the pixel of interest. In the high-resolution (BW) generation unit 39, the ratio of the W dynamic range to the B dynamic range is determined as the color change code of the pixel of interest. In the high-resolution (AW) generation unit 44, the ratio of the W dynamic range to the A dynamic range is determined as the color change code of the pixel of interest.

[0196] In step S22, the (RW) code tap selection unit 71 selects an (RW) code tap from the virtual color difference (R-W') image generated by the virtual color difference (R-W') generation unit 33. An (RW) code tap consists of, for example, multiple virtual color difference (R-W') pixels included in a predetermined peripheral region centered on a virtual color difference (R-W') pixel corresponding to a pixel of interest. The (RW) code tap selected by the (RW) code tap selection unit 71 is supplied to the (RW) code calculation unit 73. The high-resolution (GW) generation unit 38 selects a (GW) code tap. The high-resolution (BW) generation unit 39 selects a (BW) code tap. The high-resolution (AW) generation unit 44 selects an (AW) code tap.

[0197] In step S23, the (R-W) code calculation unit 73 generates the (R-W) code of the target pixel. As the method for generating the (R-W) code, the same method as that of the above-described W' code calculation unit 53, for example, DR quantization or the like can be adopted. The (R-W) code calculation unit 73 bit-combines the code generated by DR quantization or the like using the (R-W) code taps and the color change amount code supplied from the color change amount code calculation unit 70 to obtain the (R-W) code of the target pixel, and supplies it to the (R-W) coefficient storage unit 74. In the high-resolution (G-W) generation unit 38, the (G-W) code is generated. In the high-resolution (B-W) generation unit 39, the (B-W) code is generated. In the high-resolution (A-W) generation unit 44, the (A-W) code is generated.

[0198] In step S24, the (R-W) filter tap selection unit 72 selects the (R-W) filter tap of the target pixel. For example, the (R-W) filter tap selection unit 72 selects a plurality of virtual color difference (R-W') pixels included in the specified region based on the virtual color difference (R-W') pixel corresponding to the target pixel as the (R-W) filter taps. In the high-resolution (G-W) generation unit 38, the (G-W) filter tap is selected. In the high-resolution (B-W) generation unit 39, the (B-W) filter tap is selected. In the high-resolution (A-W) generation unit 44, the (A-W) filter tap is selected.

[0199] In step S25, the (R-W) coefficient storage unit 74 reads out the tap coefficient stored at the address corresponding to the (R-W) code from the already stored tap coefficients, and supplies it to the (R-W) product-sum calculation unit 75. In the high-resolution (G-W) generation unit 38, the tap coefficient corresponding to the (G-W) code is read out. In the high-resolution (B-W) generation unit 39, the tap coefficient corresponding to the (B-W) code is read out. In the high-resolution (A-W) generation unit 44, the tap coefficient corresponding to the (A-W) code is read out.

[0200] In step S26, the (R-W) product-sum operation unit 75 predicts the high-resolution pixel value (R-W) of the target pixel by performing a product-sum operation. Specifically, the (R-W) product-sum operation unit 75 performs a linear primary operation of Equation (1) using the tap coefficients supplied from the (R-W) coefficient storage unit 74 and the pixel values of the virtual color difference (R-W’) pixels supplied as (R-W) filter taps from the (R-W) filter tap selection unit 72. The predicted high-resolution pixel value (R-W) of the target pixel is supplied to the adder 40. In the high-resolution (G-W) generation unit 38, the high-resolution pixel value (G-W) of the target pixel is predicted and supplied to the adder 41. In the high-resolution (B-W) generation unit 39, the high-resolution pixel value (B-W) of the target pixel is predicted and supplied to the adder 42. In the high-resolution (A-W) generation unit 44, the high-resolution pixel value (A-W) of the target pixel is predicted and supplied to the adder 45.

[0201] In step S27, the target pixel setting unit 31 determines whether there is a pixel that has not yet been set as a target pixel for each pixel of the output image. If it is determined in step S27 that there is a pixel that has not yet been set as a target pixel, the process returns to step S20. As a result, the pixels that have not yet been set as target pixels in raster scan order are set as target pixels. On the other hand, if there is no pixel that has not yet been set as a target pixel, that is, if it is determined that all pixels of the output image have been set as target pixels, the process proceeds to step S28.

[0202] In step S28, adders 40, 41, 42, and 45 each generate an R image, a G image, a B image, and an A image, respectively, and output them to the subsequent color processing unit 18. More specifically, adder 40 generates an R image by adding a W image from the high-resolution W generation unit 36 ​​and a high-resolution (RW) image from the high-resolution (RW) generation unit 37. Adder 41 generates a G image by adding a W image from the high-resolution W generation unit 36 ​​and a high-resolution (GW) image from the high-resolution (GW) generation unit 38. Adder 42 generates a B image by adding a W image from the high-resolution W generation unit 36 ​​and a high-resolution (BW) image from the high-resolution (BW) generation unit 39. The adder 45 generates an A image by adding the W image from the high-resolution W generation unit 36 ​​and the high-resolution (AW) image from the high-resolution (AW) generation unit 44.

[0203] With the above steps completed, the demosaicing process by the demosaicing unit 17 in the second configuration example where the pixel array is a WRGAB array is finished. Through demosaicing, a W image with the same resolution as the input image or a higher resolution can be generated using only the pixel values ​​of the W pixels of the input image with a WRGAB array.

[0204] Furthermore, demosaicing interpolates and generates pixel values ​​W' equivalent to W pixels (W' pixels) at the R, G, A, and B pixel positions of the input image. Using these pixel values ​​W', virtual color differences (R-W'), (G-W'), (A-W'), and (B-W') are generated. High-resolution (RW), high-resolution (GW), high-resolution (AW), and high-resolution (BW) images are then generated from these virtual color differences. Based on the W image and the high-resolution (RW), high-resolution (GW), high-resolution (AW), and high-resolution (BW) images, the R, G, A, and B images are generated.

[0205] In other words, demosaicing generates W' pixels at the R, G, A, and B pixel positions using the W pixels, which are the most numerous pixels in the pixel array of the image sensor 12. The color difference is then calculated based on the W and W' pixels to generate the R, G, A, and B images. By calculating the color difference based on W and W' pixels, which are free of artifacts, it is possible to generate high-quality images with significantly reduced moiré, artifacts, zipper noise, false colors, color bleeding, ringing, etc. In other words, image quality degradation can be suppressed.

[0206] <10. Learning Principles of Tap Coefficients> The tap coefficients associated with the W' code, as well as the tap coefficients associated with the W code, (RW) code, (GW) code, (BW) code, and (AW) code, are learned through a training process using high-quality, high-resolution images, and contribute to the generation of high-quality images.

[0207] The tap coefficient is obtained through the training process of a code classification-type adaptive filter using the following procedure. In the training process of the code classification-type adaptive filter, the tap coefficient is obtained by using a low-resolution image (first image) as the student image and a high-resolution image (second image) as the teacher image. Here, the low-resolution image is used by, for example, reducing the image quality (resolution) of a high-resolution image by filtering it with an LPF (low-pass filter).

[0208] In the learning process, filter taps are selected from low-resolution images, and the pixel values ​​y' of pixels in high-resolution images (high-resolution pixels) are predicted using a predetermined operation, based on these filter taps and tap coefficients. This predetermined operation is, for example, a linear first-order operation. The pixel values ​​y' of high-resolution pixels can be calculated not only by linear first-order operations, but also by higher-order operations, such as second-order or higher.

[0209] If the pixel value y' of a high-resolution pixel is predicted by a linear first-order operation, the pixel value y' of a high-resolution pixel can be obtained by the linear first-order equation shown in equation (2).

[0210]

number

[0211] In equation (2), x n (N=1,2,3,···,N) is the pixel value of the nth low-resolution pixel among the N low-resolution image pixels (low-resolution pixels) that make up the filter tap. n The pixel value x of the nth low-resolution pixel is n This represents the tap coefficient that is multiplied by the value.

[0212] The true value of the pixel of the k-th sample (k-th) pixel of the training image (high-resolution image) is y. k It can be expressed as, and its true value y k The predicted value of y is obtained by equation (2). k When this is the case, the true value y k Predicted value y for k 'Prediction error e k This is expressed by equation (3).

[0213]

number

[0214] y in equation (3) k Substituting ' according to equation (2), we obtain the following equation (4).

[0215]

number

[0216] x in equation (4) n,k This is the pixel value y of the high-resolution pixel of the k-th sample. k This represents the pixel value of the nth low-resolution pixel that constitutes the filter tap for that.

[0217] Prediction error e in equation (4) k The tap coefficient w is set to 0. n However, the pixel value y of the high-resolution pixel kbecomes the optimal tap coefficient for prediction. However, for all pixel values y of high-quality pixels k finding such a tap coefficient w n is generally difficult.

[0218] Therefore, for example, the least squares method is adopted as a criterion indicating that the tap coefficient w n is the optimal value. In this case, the optimal tap coefficient w n is the coefficient when the sum E of the squared errors represented by the following equation (5) is minimized.

[0219]

Equation

[0220] K represents the number of samples in the set of pixel values x k of low-quality pixels that make up the filter taps for the pixel value y of the high-quality pixel and the pixel value y k of that high-quality pixel, x 1,k , x 2,k , ···, x n,k (the number of learning samples).

[0221] The minimum value (extreme value) of the sum E of the squared errors in equation (5) is given by w n such that the result of differentiating the sum E with respect to the tap coefficient w n is 0, as shown in equation (6).

[0222]

Equation

[0223] Therefore, differentiating the aforementioned equation (4) with respect to the tap coefficient w n yields the following equation (7).

[0224]

Equation

[0225] From equations (6) and (7), we obtain equation (8).

[0226]

number

[0227] e in equation (8) k Substituting equation (4) into equation (8), equation (8) can be expressed as the normal equation shown in equation (9).

[0228]

number

[0229] The normal equation in equation (9) can be expressed by, for example, using Gauss-Jordan elimination, which will result in the tap coefficient w n This can be solved.

[0230] By setting up and solving the normal equation (9) for each code, we can find the optimal tap coefficient (the tap coefficient that minimizes the sum of squared errors E) w n However, it is calculated for each code. The tap coefficient w is calculated in this way. n However, for example, in the case of a tap coefficient corresponding to a W' code, it is stored in the W' coefficient storage unit 54 shown in Figure 11. The tap coefficient is obtained, for example, by the learning device 100 shown in Figure 30.

[0231] <11. Example of a W' coefficient learning device configuration> Figure 30 is a block diagram showing an example configuration of a learning device that learns tap coefficients corresponding to the W' code using the method described above.

[0232] The learning device 100 in Figure 30 includes a student image generation unit 101, a teacher image generation unit 102, a pixel of interest setting unit 103, and a W' coefficient data generation unit 104.

[0233] The learning device 100 generates tap coefficients for each W' code to generate a W' image by performing learning using teacher images and student images. The learning device 100 receives a high-resolution image W having pixel values ​​of the W component as input.

[0234] The student image generation unit 101 generates a student image by down-converting and correcting the aperture area of ​​the high-resolution image W input to the learning device 100. The teacher image generation unit 102 generates a teacher image by down-converting and correcting the aperture area of ​​the high-resolution image W input to the learning device 100. The student image and the teacher image have a relationship in which the pixel positions are mutually interpolated, as shown in the relationship between the W pixels and W' pixels described above.

[0235] Figure 31 is a diagram illustrating the student and teacher images generated from the high-resolution image W.

[0236] The student image generation unit 101 generates pixel values ​​for diamond-shaped pixels with a pixel shape tilted at a 45-degree angle, which are lower resolution than the high-resolution image W, by down-converting and correcting the aperture area from the high-resolution image W. Here, the filter characteristics for down-converting and aperture area correction can be, for example, pixel-weighted averaging or LPF processing. Subsequently, the student image generation unit 101 generates an image from the down-converted and aperture-area-corrected image that stores the pixel values ​​at the same pixel positions as the W pixels of the image sensor 12, and uses this as the student image.

[0237] The teacher image generation unit 102 generates pixel values ​​for diamond-shaped pixels with a pixel shape tilted at a 45-degree angle, which are lower resolution than the high-resolution image W, by down-converting and correcting the aperture area from the high-resolution image W. Here, the filter characteristics for down-conversion and aperture area correction are the same as those of the student image generation unit 101. Subsequently, the teacher image generation unit 102 generates an image in which the pixel values ​​of the pixels other than the W pixels (W' pixels) of the image sensor 12 are stored, and this image is used as the teacher image.

[0238] The target pixel setting unit 103 in FIG. 30 sets each pixel of the teacher image generated by the teacher image generation unit 102 as a target pixel in a predetermined order.

[0239] The W' coefficient data generation unit 104 generates an optimal tap coefficient w for each code by solving the normal equation of Equation (9) for each W' code. n for each code.

[0240] <Detailed configuration example of the W' coefficient data generation unit> FIG. 32 is a diagram showing a detailed configuration example of the W' coefficient data generation unit 104.

[0241] The W' coefficient data generation unit 104 includes a W' code tap selection unit 111, a W' filter tap selection unit 112, a W' code operation unit 113, a W' normal equation addition unit 114, and a W' coefficient data operation unit 115.

[0242] The W' code tap selection unit 111, the W' filter tap selection unit 112, and the W' code operation unit 113 are each configured in the same manner as the W' code tap selection unit 51, the W' filter tap selection unit 52, and the W' code operation unit 53 shown in FIG. 11, and thus their descriptions are omitted.

[0243] The W' normal equation addition unit 114 adds, for each W' code supplied from the W' code operation unit 113, the pixel value (teacher data) y of the target pixel of the teacher image generated by the teacher image generation unit 102 k and the pixel value (student data) x of the W' filter tap for the target pixel selected by the W' filter tap selection unit 112. n,k for each W' code supplied from the W' code operation unit, the pixel value x of the filter tap is used to perform multiplication (x

[0244] between the student data in the matrix on the left side of Equation (9) n,k and an operation corresponding to summation (Σ). n,k x n’,k )

[0245] Furthermore, the W'Normal Equation Addition Unit 114 processes student data x for each W'Code supplied from the W'Code Calculation Unit 113. n,k and training data y k Using this, the student data x in the vector on the right side of equation (9) n,k and training data y k Multiplication of (x n,k y k ) and perform an operation equivalent to summation (Σ).

[0246] In other words, the W' normal equation summing unit 114 calculates the component (Σx) of the matrix on the left side of equation (9) obtained for the training data previously identified as the pixel of interest. n,k x n’,k ) and the component of the vector on the right side (Σx n,k y k ) and are stored in a memory unit not shown. Then, the W'normal equation addition unit 114 calculates the matrix component (Σx n,k x n’,k ) or vector component (Σx n,k y k Regarding the newly designated pixels of interest in the training data, the training data y k+1 and student data x n,k+1 The corresponding component x is calculated using n,k+1 x n’,k+1 or x n,k+1 y k+1 Add them together (perform the addition shown in equation (9)).

[0247] Then, the W' normal equation summing unit 114 uses all of the input high-resolution image training data as the pixels of interest and performs the aforementioned summation to establish the normal equation shown in equation (9) for each W' code. The W' normal equation summing unit 114 supplies the normal equation for each W' code to the W' coefficient data calculation unit 115.

[0248] The W' coefficient data calculation unit 115 solves the normal equation for each W' code supplied from the W' normal equation addition unit 114, thereby determining the optimal tap coefficient w for each W' code. nThe W' coefficient data calculation unit 115 then calculates the obtained tap coefficient w according to the type of training image (W' component image) for which the pixel of interest is set. n This is output as a tap coefficient for generating the pixel value W' of the W' pixel. The tap coefficients for each W' code obtained in this way are stored in the W' coefficient storage unit 54 shown in Figure 11.

[0249] <12. Flowchart of the learning process for the W' coefficient> Referring to the flowchart in Figure 33, the tap coefficient w for each W' code by the learning device 100 n This section describes the learning process for learning the (W' coefficient). This learning process is initiated, for example, when a high-resolution image W is input.

[0250] First, in step S31, the student image generation unit 101 generates a student image (student data) by down-converting and correcting the aperture area of ​​the input high-resolution image W. The student image is lower resolution than the high-resolution image W, has a diagonal grid-like pixel arrangement, and is composed of an image that stores the pixel values ​​at the same pixel positions as the W pixels of the image sensor 12.

[0251] In step S32, the training image generation unit 102 generates a training image (training data) from the high-resolution image W by down-converting and correcting the aperture area. The training image has a lower resolution than the high-resolution image W, and consists of an image with a diagonal grid-like pixel arrangement that stores the pixel values ​​of pixels at the same pixel positions as the pixels other than the W pixels (W' pixels) of the image sensor 12.

[0252] In step S33, the focus pixel setting unit 103 sets pixels that have not yet been set as focus pixels among the pixels that make up the generated training image as focus pixels. For example, the focus pixel setting unit 103 sets pixels that have not yet been set as focus pixels in the raster scan order among the pixels that make up the training image as focus pixels.

[0253] In step S34, the W' code tap selection unit 111 selects a W' code tap for the pixel of interest from the student image. Similar to the W' code tap selection unit 51, the W' code tap selection unit 111 selects multiple pixels included in a predetermined surrounding area centered on the pixel of interest in the student image as W' code taps.

[0254] In step S35, the W' code calculation unit 113 extracts features based on the W' code tap and generates the W' code for the pixel of interest. For example, as explained with reference to Figure 15, the W' code can be generated using, for example, DR quantization.

[0255] In step S36, the W' filter tap selection unit 112 selects a W' filter tap for the pixel of interest from the student image. Similar to the W' filter tap selection unit 52, the W' filter tap selection unit 112 selects multiple pixels included in a predetermined surrounding area centered on the pixel of interest in the student image as W' filter taps.

[0256] In step S37, the W' normal equation summing unit 114 generates a linear equation shown in equation (2) above, and the W' filter tap selected by the W' filter tap selection unit 112 is used for pixels x1, x2, ..., x in equation (2). n It is used as follows. The W' normal equation addition unit 114 adds the linear equations generated in this way for each W' code generated in step S35 to generate the normal equation of equation (9). The W' normal equation addition unit 114 supplies the generated normalized equation to the W' coefficient data calculation unit 115.

[0257] In step S38, the focus pixel setting unit 103 determines whether there are any pixels in the training image that have not yet been set as focus pixels. If it is determined in step S38 that there are pixels that have not yet been set as focus pixels, the process returns to step S33. As a result, the process from step S33 to step S38 described above is repeated for each new focus pixel.

[0258] On the other hand, if it is determined in step S38 that all pixels constituting the training image have been set as pixels of interest, the process proceeds to step S39. In step S39, the W' coefficient data calculation unit 115 calculates the optimal tap coefficient w for each W' code. n The (W' coefficient) is generated. Specifically, the W' coefficient data calculation unit 115 uses, for example, Gauss-Jordan elimination to calculate the tap coefficient w for the normal equation in equation (9). n The tap coefficient w is calculated for each W' code. n This is supplied to and stored in the W' coefficient storage unit 54 shown in Figure 11.

[0259] This completes the learning process for learning the W' coefficient.

[0260] Furthermore, in order to reduce the processing load in the interpolation W' generation unit 32, when generating tap coefficients that are commonly used for any input image, the W' coefficient data generation unit 104 in Figure 32 consists only of the W' filter tap selection unit 112, the W' normal equation addition unit 114, and the W' coefficient data calculation unit 115. In the W' filter tap selection unit 112, multiple pixels around the pixel of interest are selected as W' filter taps, and a common tap coefficient w is applied to all W' codes. n In other words, the tap coefficient w regardless of the W' chord. n However, it is generated by the W' coefficient data calculation unit 115.

[0261] <13. Example configuration of a learning device corresponding to the first configuration example (WRGB array)> Figure 34 is a block diagram showing an example configuration of a learning device corresponding to the first configuration example of the demosaicing section 17.

[0262] The learning device 120 in Figure 34 generates tap coefficients used to generate high-resolution W images, high-resolution (RW) images, high-resolution (GW) images, and high-resolution (BW) images, respectively, by learning using teacher images and student images. Hereinafter, the tap coefficients used to generate high-resolution W images, high-resolution (RW) images, high-resolution (GW) images, and high-resolution (BW) images will be referred to as the W coefficient, (RW) coefficient, (GW) coefficient, and (BW) coefficient, respectively.

[0263] The learning device 120 includes a student image generation unit 101, a teacher image generation unit 102, a pixel of interest setting unit 103, an interpolation W' generation unit 32, a virtual color difference (R-W') generation unit 33, a virtual color difference (G-W') generation unit 34, and a virtual color difference (B-W') generation unit 35. The learning device 120 also includes a W coefficient data generation unit 121, an (RW) coefficient data generation unit 122, a (GW) coefficient data generation unit 123, and a (BW) coefficient data generation unit 124.

[0264] The learning device 120 receives, for example, four high-resolution images (4 channels) generated from four image sensors corresponding to the W, R, G, and B components, respectively. The four high-resolution images are high-resolution images of the W, R, G, and B components.

[0265] The student image generation unit 101 generates student images by downconverting and correcting the aperture area using high-resolution images of the W, R, G, and B components. The teacher image generation unit 102 generates teacher images of the W, R, G, and B components by downconverting the high-resolution images of the W, R, G, and B components, respectively.

[0266] Figure 35 illustrates the student and teacher images generated from high-resolution images of the W, R, G, and B components.

[0267] The student image generation unit 101 performs downconversion and aperture area correction on the high-resolution images of the W, R, G, and B components, respectively, to generate pixel values ​​for rhombus pixels in a diagonal grid pixel arrangement at a lower resolution than the high-resolution images. Then, the student image generation unit 101 generates a diagonal grid WRGB array image using the downconverted and aperture area corrected low-resolution images of the W, R, G, and B components, thereby generating a student image. The generated student image corresponds to the image output directly from the WRGB array image sensor 12. The filter characteristics for downconversion and aperture area correction can be, for example, pixel-weighted averaging or LPF processing.

[0268] The teacher image generation unit 102 downconverts the high-resolution images of the W, R, G, and B components respectively to generate low-resolution images of the W, R, G, and B components. The generated low-resolution images have a lower resolution than the high-resolution images input to the learning device 120, but have the same resolution as the output images output by the demosaicing unit 17. The filter characteristics for downconversion differ from those of the student image generation unit 101, and include pixel averaging and LPF processing.

[0269] The pixel of interest setting unit 103 in Figure 34 sets each pixel of the training image generated by the training image generation unit 102 as a pixel of interest in a predetermined order. The pixel of interest setting unit 103 supplies information that identifies the pixels set as pixels of interest to each block, such as the interpolation W' generation unit 32, the virtual color difference (R-W') generation unit 33, the W coefficient data generation unit 121, and the (RW) coefficient data generation unit 122.

[0270] The interpolation W' generation unit 32, the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, and the virtual color difference (B-W') generation unit 35 are the same as the demosaicing unit 17 described above, so their explanations will be omitted.

[0271] The W coefficient data generation unit 121 sets up and solves the normal equation (9) for each W code to obtain the optimal W coefficient w n Calculate this for each W code.

[0272] The (RW) coefficient data generation unit 122 solves the normal equation (9) for each (RW) code to obtain the optimal (RW) coefficient w n Calculate this for each W code.

[0273] The (GW) coefficient data generation unit 123 sets up and solves the normal equation (9) for each (GW) code to determine the optimal (GW) coefficient w n Calculate this for each W code.

[0274] The (BW) coefficient data generation unit 124 sets up and solves the normal equation (9) for each (BW) code to obtain the optimal (BW) coefficient w n Calculate this for each W code.

[0275] <14. Example configuration of a learning device corresponding to the second configuration example (WRGAB array)> Figure 36 is a block diagram showing an example configuration of a learning device corresponding to the second configuration example of the demosaicing unit 17.

[0276] In Figure 36, parts that are common to the configuration of the learning device shown in Figure 34 are denoted by the same reference numerals, and explanations of those parts are omitted as appropriate.

[0277] The learning device 130 in Figure 36 includes a student image generation unit 101, a teacher image generation unit 102, a pixel of interest setting unit 103, an interpolation W' generation unit 32, a virtual color difference (R-W') generation unit 33, a virtual color difference (G-W') generation unit 34, a virtual color difference (B-W') generation unit 35, and a virtual color difference (A-W') generation unit 43. The learning device 120 also includes a W coefficient data generation unit 121, an (RW) coefficient data generation unit 122, a (GW) coefficient data generation unit 123, a (BW) coefficient data generation unit 124, and an (AW) coefficient data generation unit 131.

[0278] Therefore, in the learning device 130 of Figure 36, a virtual color difference (A-W') generation unit 43 and an (AW) coefficient data generation unit 131 are newly added as A pixels are added to the pixel array.

[0279] The learning device 130 generates tap coefficients used to generate high-resolution W images, high-resolution (RW) images, high-resolution (GW) images, high-resolution (BW) images, and high-resolution (AW) images, respectively, by learning using teacher images and student images. Hereinafter, the tap coefficient used to generate high-resolution (AW) images will be referred to as the (AW) coefficient.

[0280] The learning device 120 receives, for example, five high-resolution images (5 channels) generated from five image sensors, each corresponding to the W, R, G, B, and A components. The five high-resolution images are high-resolution images of the W, R, G, B, and A components.

[0281] The student image generation unit 101 generates student images by downconverting and correcting the aperture area using high-resolution images of the W, R, G, B, and A components. The teacher image generation unit 102 generates teacher images of the W, R, G, B, and A components by downconverting the high-resolution images of the W, R, G, B, and A components, respectively.

[0282] Figure 37 illustrates the student and teacher images generated from high-resolution images of the W, R, G, B, and A components.

[0283] The student image generation unit 101 performs down-conversion and aperture area correction on the high-resolution images of the W component, R component, G component, B component, and A component respectively, and generates pixel values of rhombic pixels in a diagonal lattice pixel array at a lower resolution than the high-resolution images. Then, the student image generation unit 101 generates a student image by generating an image of a diagonal lattice WRGAB array using the low-resolution images of the W component, R component, G component, B component, and A component on which down-conversion and aperture area correction have been performed. The generated student image corresponds to the image output directly from the image sensor 12 of the WRGAB array. The filter characteristics of the down-conversion and aperture area correction may be, for example, pixel weighted average processing, LPF processing, etc.

[0284] The teacher image generation unit 102 performs down-conversion on the high-resolution images of the W component, R component, G component, B component, and A component respectively, and generates low-resolution images of the W component, R component, G component, B component, and A component. The generated low-resolution images are lower in resolution than the high-resolution images input to the learning device 130, but have the same resolution as the output image output by the demosaicing unit 17. The filter characteristics of the down-conversion are different from those of the student image generation unit 101, and are pixel average processing, LPF processing, etc.

[0285] The virtual chromatic aberration (A-W’) generation unit 43 is the same as the above-described demosaicing unit 17, and thus its description is omitted.

[0286] (A-W) coefficient data generation unit 131 solves the normal equation of Equation (9) for each (A-W) code, and obtains the optimal (A-W) coefficient w n for each (A-W) code.

[0287] <Detailed configuration example of W coefficient data generation unit> FIG. 38 is a diagram showing a detailed configuration example of the W coefficient data generation unit 121.

[0288] The W coefficient data generation unit 107 includes a W code tap selection unit 141, a W filter tap selection unit 142, a W code calculation unit 143, a W normal equation addition unit 144, and a W coefficient data calculation unit 145.

[0289] The W code tap selection unit 141, the W filter tap selection unit 142, and the W code calculation unit 143 are configured in the same manner as the W code tap selection unit 61, the W filter tap selection unit 62, and the W code calculation unit 63 shown in FIG. 16, and thus their descriptions are omitted.

[0290] For each W code supplied from the W code calculation unit 143, the W normal equation addition unit 144 adds the pixel value (teacher data) y of the target pixel of the teacher image generated by the teacher image generation unit 102 k and the pixel value (student data) x of the W filter tap for the target pixel selected by the W filter tap selection unit 142. n,k And it performs addition for them.

[0291] Then, for each W code supplied from the W code calculation unit 143, the W normal equation addition unit 144 uses the student data x n,k to perform multiplication of the student data with each other in the matrix on the left side of Equation (9) (x n,k x n’,k ) and an operation corresponding to summation (Σ).

[0292] Furthermore, for each W code supplied from the W code calculation unit 143, the W normal equation addition unit 144 uses the student data x n,k and the teacher data y k to perform multiplication of the student data x n,k and the teacher data y k in the vector on the right side of Equation (9) (x n,k y k ) and an operation corresponding to summation (Σ).

[0293] That is, the W normal equation addition unit 144 is the component of the matrix on the left side in Equation (9) obtained for the teacher data regarded as the previous target pixel (Σx n,k x n’,k) and the components of the vector on the right side (Σx n,k y k ) are stored in a memory unit not shown. Then, the W normal equation addition unit 144 calculates, for the matrix component (Σx n,k x n’,k ) or the vector component (Σx n,k y k ), the corresponding component x k+1 or x n,k+1 and the student data x n,k+1 x n’,k+1 or x n,k+1 y k+1 for the teacher data newly regarded as the target pixel, using the teacher data y k+1 and the student data x n,k+1 (performing the addition represented by the summation in Equation (9)).

[0294] Then, the W normal equation addition unit 144 forms the normal equation shown in Equation (9) for each W code by performing the above-mentioned addition with all the teacher data of the input high-resolution image regarded as the target pixel. The W normal equation addition unit 144 supplies the normal equation of each W code to the W coefficient data calculation unit 145.

[0295] The W coefficient data calculation unit 145 calculates the optimal tap coefficient w n for each W code by solving the normal equation of each W code supplied from the W normal equation addition unit 144. Then, the W coefficient data calculation unit 145 outputs the obtained tap coefficient w n as the W coefficient required for performing the filter operation according to the type of the teacher image (W component image) in which the target pixel is set. The W coefficient for each W code thus obtained is stored in the W coefficient storage unit 64 shown in FIG. 16.

[0296] <Detailed Configuration Example of (R - W) Coefficient Data Generation Unit> FIG. 39 is a diagram showing a configuration example of the (R - W) coefficient data generation unit 122.

[0297] The (RW) coefficient data generation unit 122 includes a color change amount code calculation unit 150, an (RW) code tap selection unit 151, an (RW) filter tap selection unit 152, an (RW) teacher generation unit 153, an (RW) code calculation unit 154, an (RW) normal equation addition unit 155, and an (RW) coefficient data calculation unit 156.

[0298] The color change amount code calculation unit 150, the (RW) code tap selection unit 151, the (RW) filter tap selection unit 152, and the (RW) code calculation unit 154 are configured in the same way as the color change amount code calculation unit 70, the (RW) code tap selection unit 71, the (RW) filter tap selection unit 72, and the (RW) code calculation unit 73 shown in Figure 19, so their descriptions will be omitted.

[0299] The (RW) teacher generation unit 153 generates an (RW) component, which is a real color difference rather than a virtual color difference, by calculating the difference between the R component and the W component of the teacher image generated by the teacher image generation unit 102 at the R pixel position of the pixel array which is a WRGB array or a WRGAB array. The (RW) teacher generation unit 153 supplies the generated (RW) component as teacher data to the (RW) normal equation addition unit 155.

[0300] The (RW) Normal Equation Addition Unit 155 adds the color difference component (RW) (training data) y of the pixel of interest generated by the (RW) Teacher Generation Unit 153 for each (RW) code supplied from the (RW) Code Calculation Unit 154. k And, (RW) filter tap (student data) x for the pixel of interest selected by the (RW) filter tap selection unit 152 n,k Perform an addition targeting the following.

[0301] In other words, the (RW) normal equation addition unit 155 performs the same processing as the W normal equation addition unit 144, except that it uses the (RW) component instead of the W component of the training image as training data, and sets up the normal equation shown in equation (9) for each (RW) code. Then, the (RW) normal equation addition unit 155 supplies the normal equation for each (RW) code to the (RW) coefficient data calculation unit 156.

[0302] The (RW) coefficient data calculation unit 156 generates the (RW) coefficient by performing the same processing as the W coefficient data calculation unit 145.

[0303] <15. Flowchart for learning the W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient> Next, referring to the flowchart in Figure 40, we will explain the learning process for learning the W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient, which correspond to the case where the pixel array is a WRGAB array. This learning process starts, for example, when a 5-channel high-resolution image of the W, R, G, B, and A components is input.

[0304] In step S41, the student image generation unit 101 generates a student image by performing downconversion and aperture area correction using high-resolution images of the W, R, G, B, and A components.

[0305] In step S42, the training image generation unit 102 generates training images for the W, R, G, B, and A components by downconverting the high-resolution images of the W, R, G, B, and A components, respectively.

[0306] In step S43, the focus pixel setting unit 103 sets as focus pixels any pixels that have not yet been set as focus pixels among the pixels that make up the generated training image. For example, the focus pixel setting unit 103 sets as focus pixels any pixels that have not yet been set as focus pixels in the raster scan order among the pixels that make up the training image.

[0307] In step S44, the W' code tap selection unit 51 of the interpolation W' generation unit 32 selects a W' code tap for the pixel of interest from the student image.

[0308] In step S45, the W' code calculation unit 53 of the interpolation W' generation unit 32 extracts feature quantities based on the W' code tap and generates the W' code of the pixel of interest.

[0309] In step S46, the W' filter tap selection unit 52 of the interpolation W' generation unit 32 selects a W' filter tap for the pixel of interest from the student image.

[0310] In step S47, the W' coefficient storage unit 54 of the interpolation W' generation unit 32 reads the tap coefficients stored at the address corresponding to the W' code supplied from the W' code calculation unit 53 and supplies them to the W' sum-of-products calculation unit 55.

[0311] In step S48, the W' sum-of-products calculation unit 55 of the interpolation W' generation unit 32 generates (predicts) a pixel value W' for the pixel of interest by performing a sum-of-products calculation. Specifically, the W' sum-of-products calculation unit 55 performs a linear first-order calculation of equation (1) using the tap coefficient supplied from the W' coefficient storage unit 54 and the pixel value W of the W' filter tap supplied from the W' filter tap selection unit 52. The generated pixel value W' is supplied to the virtual color difference (R-W') generation unit 33, the virtual color difference (G-W') generation unit 34, the virtual color difference (B-W') generation unit 35, and the virtual color difference (A-W') generation unit 43.

[0312] In step S49, the focus pixel setting unit 103 determines whether there are any pixels that have not yet been set as focus pixels in order to generate the W' pixel. If it is determined in step S38 that there are pixels that have not yet been set as focus pixels, the process returns to step S43. As a result, the process from step S43 to step S49 described above is repeated for the new focus pixels.

[0313] On the other hand, if it is determined in step S49 that all pixels that generate the W' pixels have been set as pixels of interest, the process proceeds to step S50.

[0314] In step S50, a virtual color difference (R-W'), virtual color difference (G-W'), virtual color difference (B-W'), and virtual color difference (A-W') image are generated. This process is the same as the process in step S18 in the demosaicing process shown in Figure 29. The generated virtual color difference (R-W') image is supplied to the (RW) coefficient data generation unit 122, and the virtual color difference (G-W') image is supplied to the (GW) coefficient data generation unit 123. The virtual color difference (B-W') image is supplied to the (BW) coefficient data generation unit 124, and the virtual color difference (A-W') image is supplied to the (AW) coefficient data generation unit 131.

[0315] In step S51, the focus pixel setting unit 103 sets pixels that have not yet been designated as focus pixels among the pixels that make up the training image generated by the training image generation unit 102 as focus pixels. For example, the focus pixel setting unit 103 sets pixels that have not yet been designated as focus pixels among the pixels that make up the training image in raster scan order as focus pixels.

[0316] In step S52, the (RW) coefficient data generation unit 122's (RW) teacher generation unit 153 generates the (RW) component, which is the actual color difference rather than the virtual color difference, by calculating the difference between the R and W components of the teacher image. Similarly, the (GW) coefficient data generation unit 123, the (BW) coefficient data generation unit 124, and the (AW) coefficient data generation unit 131 generate the (GW), (BW), and (AW) components, which are the actual color differences.

[0317] In step S53, the color change amount code calculation unit 150 of the (RW) coefficient data generation unit 122 calculates and determines the color change amount code of the pixel of interest and outputs it to the (RW) code calculation unit 154. The processing in step S53 is the same as the processing in step S21 in the demosaicing process shown in Figure 29. Similarly, the color change amount code is calculated in the (GW) coefficient data generation unit 123, the (BW) coefficient data generation unit 124, and the (AW) coefficient data generation unit 131.

[0318] In step S54, a code tap for the pixel of interest is selected. The W code tap selection unit 141 shown in Figure 38 selects the W code tap for the pixel of interest from the student image. The (RW) code tap selection unit 151 of the (RW) coefficient data generation unit 122 shown in Figure 39 selects the (RW) code tap for the pixel of interest from the virtual color difference (R-W') image. Similarly, the (GW) coefficient data generation unit 123, the (BW) coefficient data generation unit 124, and the (AW) coefficient data generation unit 131 select the (GW) code tap, (BW) code tap, and (AW) code tap, respectively.

[0319] In step S55, the W code, (RW) code, (GW) code, (BW) code, and (AW) code are generated using the code tap of the pixel of interest. The W code calculation unit 143 of the W coefficient data generation unit 121 shown in Figure 38 extracts feature quantities based on the supplied W code tap and generates the W code of the pixel of interest. The (RW) code calculation unit 154 of the (RW) coefficient data generation unit 122 shown in Figure 39 generates the (RW) code of the pixel of interest by bit-combining the code generated based on the (RW) code tap from the (RW) code tap selection unit 151 and the color change amount code from the color change amount code calculation unit 150. The (GW) coefficient data generation unit 123 generates the (GW) code of the pixel of interest by bit-combining the code generated based on the (GW) code tap and the color change amount code. The (BW) coefficient data generation unit 124 generates the (BW) code of the pixel of interest by bit-combining the code generated based on the (BW) code tap and the color change amount code. The (AW) coefficient data generation unit 131 generates the (AW) code of the pixel of interest by bit-combining the code generated based on the (AW) code tap with the color change amount code.

[0320] In step S56, a filter tap for the pixel of interest is selected. The W filter tap selection unit 142 of the W coefficient data generation unit 121 shown in Figure 38 selects the W filter tap for the pixel of interest from the student image. The (RW) filter tap selection unit 152 of the (RW) coefficient data generation unit 122 shown in Figure 39 selects the (RW) filter tap for the pixel of interest from the virtual color difference (R-W') image. The (GW) coefficient data generation unit 123 selects the (GW) filter tap for the pixel of interest from the virtual color difference (G-W') image. The (BW) coefficient data generation unit 124 selects the (BW) filter tap for the pixel of interest from the virtual color difference (B-W') image. The (AW) coefficient data generation unit 131 selects the (AW) filter tap for the pixel of interest from the virtual color difference (A-W') image.

[0321] In step S57, for each W code, (RW) code, (GW) code, (BW) code, and (AW) code, the normalization equations generated based on the linear equation (2) described above are added together. The W normal equation addition unit 144 of the W coefficient data generation unit 121 shown in Figure 38 generates the linear equation shown in equation (2) described above, and the W filter taps selected by the W filter tap selection unit 142 are used for pixels x1, x2, ..., x of equation (2). n It is used as such. The W normal equation addition unit 144 adds the linear equations generated in this way to each code generated in step S55 to generate the normal equation of equation (9). The W normal equation addition unit 144 supplies the generated normalized equation to the W coefficient data calculation unit 145. The (RW) normal equation addition unit 155, (GW) coefficient data generation unit 123, (BW) coefficient data generation unit 124, and (AW) coefficient data generation unit 131 of the (RW) coefficient data generation unit 122 shown in Figure 39 similarly generate normalized equations.

[0322] In step S58, the focus pixel setting unit 103 determines whether there are any pixels in the training image that have not yet been set as focus pixels. If it is determined in step S58 that there are pixels that have not yet been set as focus pixels, the process returns to step S51. As a result, the process from step S51 to step S58 described above is repeated for each new focus pixel.

[0323] On the other hand, if it is determined in step S58 that all pixels constituting the training image have been set as pixels of interest, the process proceeds to step S59. In step S59, the W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient are generated for each code. Specifically, the W coefficient data calculation unit 145 of the W coefficient data generation unit 121 shown in Figure 38 generates the W coefficient for each W code by, for example, using Gaussian elimination on the normal equation of equation (9). The (RW) coefficient data calculation unit 156 of the (RW) coefficient data generation unit 122 shown in Figure 39 generates the (RW) coefficient for each (RW) code. The (GW) coefficient data generation unit 123 generates the (GW) coefficient. The (BW) coefficient data generation unit 124 generates the (BW) coefficient for each (BW) code. The (AW) coefficient data generation unit 131 generates the (AW) coefficient for each (AW) code. The W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient for each generated code are stored in the high-resolution W generation unit 36, high-resolution (RW) generation unit 37, high-resolution (GW) generation unit 38, high-resolution (BW) generation unit 39, and high-resolution (AW) generation unit 44, respectively.

[0324] This completes the learning process for acquiring the W coefficient, (RW) coefficient, (GW) coefficient, (BW) coefficient, and (AW) coefficient.

[0325] Furthermore, to reduce processing load, when generating tap coefficients that are commonly used for any input image, the W coefficient data generation unit 121 in Figure 38 consists of a W filter tap selection unit 142, a W normal equation addition unit 144, and a W coefficient data calculation unit 145. The (RW) coefficient data generation unit 122 in Figure 39 consists only of a (RW) filter tap selection unit 152, a (RW) teacher generation unit 153, a (RW) normal equation addition unit 155, and a (RW) coefficient data calculation unit 156. Tap coefficients w, regardless of whether they are W code or (RW) code. n This is generated by the learning process. The same applies to the (GW) coefficient data generation unit 123, the (BW) coefficient data generation unit 124, and the (AW) coefficient data generation unit 131.

[0326] <16. Example of Optical System Configuration> Figure 41 shows an example of the configuration of the optical system 11.

[0327] The optical system 11 includes at least an imaging lens 161 and an optical low-pass filter 162. The optical low-pass filter 162 is provided between the imaging lens 161 and the image sensor 12. Light reflected from an object (not shown) and incident on the imaging lens 161 reaches the image sensor 12 via the optical low-pass filter 162. The optical low-pass filter 162 imparts a low-pass filter effect that affects the imaging performance of the optical system 11. The image sensor 12 performs photoelectric conversion (imaging) of the subject image (optical image) formed by the light that has passed through the optical low-pass filter 162 and to which the low-pass filter effect has been applied.

[0328] Figure 42 shows an example configuration of the optical low-pass filter 162.

[0329] The optical low-pass filter 162 is constructed by stacking four rectangular birefringent optical elements 162a to 162d, each having the same shape as the light-receiving area of ​​the image sensor 12. The characteristics of the optical low-pass filter 162 are realized by the combination of these four birefringent optical elements 162a to 162d.

[0330] As shown in Figure 42, the axis extending in the direction of the long side (horizontal direction) of the image sensor 12 is defined as the x-axis, and its direction is defined as the x-direction. The axis extending in the direction of the short side (vertical direction) of the image sensor 12 is defined as the y-axis, and its direction is defined as the y-direction. The plane along the x-axis and y-axis is called the xy-plane. The optical axis is an axis perpendicular to the xy-plane, and the direction in which the optical axis extends is defined as the optical axis direction.

[0331] The birefringent optical elements 162a to 162d have the structure shown in Figure 43. Each birefringent optical element 162a to 162d is made of a uniaxial crystal such as quartz or lithium niobate, and the optical axis of the uniaxial crystal is positioned at an inclination with respect to the xy plane. As shown in Figure 43A, when the birefringent optical elements 162a to 162d are viewed from above the xy plane (in the direction of the optical axis), the optical axis of the uniaxial crystal is oriented at an angle of φ = 90° with respect to the y axis.

[0332] As shown in Figure 43B, when a single incident ray 164 is incident on the birefringent optical elements 162a to 162d from the optical axis direction, the ordinary ray 165 is transmitted straight through the birefringent optical elements 162a to 162d, and the extraordinary ray 166 is separated from the ordinary ray 165 by the birefringent optical elements 162a to 162d in the direction shown with a separation width D. The ordinary ray 165 becomes polarized in a direction perpendicular to the separation direction between the ordinary ray 165 and the extraordinary ray 166, and the extraordinary ray 166 becomes polarized in the separation direction.

[0333] Figure 44A shows an example of the pixel arrangement of the image sensor 12, which is the WRGAB arrangement described above. Figure 44B shows the direction of light ray separation of the four birefringent optical elements 162a to 162d that constitute the optical low-pass filter 162 in relation to the direction of the image sensor 12. The angle is defined as a clockwise direction around the y-axis. Birefringent optical element 162a shows that, with respect to light rays incident from the optical axis direction, ordinary rays with polarization in the x-direction pass straight through the element, and extraordinary rays with polarization in the y-direction are separated at an angle of 45° to the ordinary rays.

[0334] Similarly, the birefringent optical element 162b separates at an angle of 225°, 180° opposite to that of the birefringent optical element 162a, the birefringent optical element 162c separates at an angle of 135°, and the birefringent optical element 162d separates at an angle of 315°. The separation directions of the birefringent optical elements 162a to 162d all correspond to the direction of the pixel array of the image sensor 12, in other words, a 45-degree oblique direction.

[0335] Figure 45A shows the light-gathering area by the on-chip microlenses formed on the image sensor 12 and the ideal light-gathering area of ​​the generated pixels when the optical low-pass filter 162 is not provided.

[0336] The area inside the white circle within the W pixel indicates the focusing area of ​​the on-chip microlens of the image sensor 12. In contrast to the W pixel showing the focusing area of ​​the on-chip microlens, the ideal focusing area of ​​the generated pixel is shown by the rectangular dotted line area. Currently, the focusing area of ​​the on-chip microlens does not cover the focusing area of ​​the generated pixel.

[0337] Figure 45B shows the light-gathering area by the on-chip microlenses formed on the image sensor 12 and the ideal light-gathering area of ​​the generated pixels when an optical low-pass filter 162 is provided.

[0338] By designing the separation width D of the optical low-pass filter 162 to be approximately half the diagonal grid-like pixel pitch p of the image sensor 12, the light-gathering area of ​​the on-chip microlens can cover the ideal light-gathering area of ​​the generated pixels. Furthermore, since there is little overlap in the horizontal and vertical directions, the problem of resolution degradation in the generated pixels is minimal.

[0339] In the interpolation W' generation unit 32, which generates pixel values ​​W' equivalent to W pixels at the R, G, A, and B pixel positions of the input image, the pixel values ​​W' are generated using only the W pixels surrounding the generated pixels, thereby improving the prediction accuracy of the pixel values ​​W'.

[0340] Figure 45 illustrates the effect of the optical low-pass filter 162 on W pixels, but the optical low-pass filter 162 similarly focuses light on other color pixels, such as R pixels. As a result, for example, the light from the W pixels is mixed with the light from the R pixels.

[0341] In generating student and teacher images in Figures 31, 35, and 37, the accuracy of predicting pixel values ​​W' can be further improved by training the characteristics of the optical low-pass filter 162, thereby suppressing false colors and moiré patterns.

[0342] Therefore, by providing an optical system 11 having the characteristics of the optical low-pass filter 162 described above, the imaging device 10 can suppress the occurrence of false colors and moiré patterns in the vicinity of the Nyquist frequency and in higher spatial frequency ranges, regardless of the F-number or incident position of the incident light beam.

[0343] <17. Example of pixel size configuration> Figure 46 shows the spectral sensitivity characteristics of pixels in the image sensor 12 that have each color filter.

[0344] As shown in Figure 46, the relative sensitivity differs depending on the W, R, G, and B color filters, resulting in different exposure amounts per unit time. In other words, due to differences in spectral sensitivity characteristics, it is possible that pixels of one color may be saturated while pixels of another color are not exposed to their ideal state.

[0345] Figure 47 shows a first example configuration of the photodiode and on-chip microlens of the image sensor 12 when the pixel array is a WRGAB array.

[0346] In Figure 47, the WRGAB array is a zigzag pixel arrangement in which achromatic pixels (W pixels) and chromatic pixels (R pixels, G pixels, B pixels, and A pixels) are arranged in a checkerboard pattern. When the pixel arrangement is a WRGB array, the A pixels also become G pixels. The area inside the white circle shown within the diamond-shaped pixels indicates the light-gathering area by the on-chip microlens. In the pixel arrangement of Figure 47, the pixel size and the size of the photodiode are the same for both achromatic and chromatic pixels.

[0347] Figure 48 shows the time it takes for W, R, G, and B pixels to reach saturation exposure when the size of the photodiodes is the same for achromatic and chromatic pixels, as shown in the pixel arrangement in Figure 47.

[0348] In Figure 48, the horizontal axis represents exposure time, and the vertical axis represents exposure amount. 100% exposure corresponds to saturation exposure.

[0349] As can be seen from Figure 48, the slope of the exposure curve QW for W pixels is the steepest. W pixels receive a large amount of exposure per unit time and reach saturation exposure at time tW. The slope of the exposure curve QR for R pixels is the second steepest after W pixels, and R pixels reach saturation exposure at time tR (>tW). The slope of the exposure curve QG for G pixels is the third steepest after R pixels, and G pixels reach saturation exposure at time tG (>tR). The slope of the exposure curve QB for B pixels is the smallest, and B pixels reach saturation exposure at time tB (tG). Thus, when the size of the photodiode is the same for achromatic and chromatic pixels, the time it takes to reach saturation exposure differs significantly between W pixels and R, G, and B pixels.

[0350] Figure 49 shows a second example configuration of the photodiode and on-chip microlens of the image sensor 12 when the pixel array is a WRGAB array.

[0351] In the pixel array shown in Figure 49, the achromatic pixels (W pixels) have an octagonal shape, and their pixel size and photodiode size are larger than those of the chromatic pixels. The area ratio of achromatic pixels to chromatic pixels is determined, for example, by the ratio of the transmittances of the achromatic filter and the chromatic filter. However, the size of the on-chip microlenses is the same for both achromatic and chromatic pixels.

[0352] Figure 50 shows the time it takes for W pixels, R pixels, G pixels, and B pixels to reach saturation exposure, when the pixel sizes of achromatic pixels and chromatic pixels are set to an area ratio corresponding to the ratio of transmittances of the achromatic filter and the chromatic filter, as shown in the pixel array in Figure 49.

[0353] As can be seen from Figure 50, the slope of the exposure curve QW for achromatic pixels (W pixels) and the slope of the exposure curve QR for R pixels are almost the same and are the largest. The slopes of the exposure curve QG for G pixels and the exposure curve QB for B pixels are also close to the slopes of exposure curves QW and QR, and the time to reach saturation exposure does not differ significantly among W pixels, R pixels, G pixels, and B pixels compared to Figure 48. In other words, the junction capacitance is large in achromatic pixels, so the time to saturation is longer, while the junction capacitance is small in chromatic pixels, so the time to saturation is shorter. As a result, the saturation times for achromatic and chromatic pixels can be made closer.

[0354] In the image sensor 12, it is preferable to have a pixel array in which the pixel size (photodiode size) of the achromatic pixels is larger than that of the chromatic pixels. However, the size of the on-chip microlenses may be the same for both the achromatic and chromatic pixels.

[0355] Figure 51 shows a third example configuration of the photodiode and on-chip microlens of the image sensor 12 when the pixel array is a WRGAB array.

[0356] In the third configuration example, the pixel size of the achromatic pixels (W pixels) is set to be the size obtained by dividing the octagonal achromatic pixels of the second configuration example shown in Figure 49 into four parts. In other words, one W pixel in the second configuration example shown in Figure 49 corresponds to four W pixels in the third configuration example. The area ratio between the pixel size of the octagonal shape composed of four W pixels and the pixel size of the chromatic pixels is, as in the second configuration example, corresponding to the ratio of the transmittance of the achromatic filter and the chromatic filter. The size of the on-chip microlenses is the same for the four W pixels and the chromatic pixels.

[0357] The resolution of the W pixels is four times that of the second configuration example shown in Figure 49. When calculating the color difference, the pixel value W of the W pixels is the average value of the four W pixels that make up the octagon. The third configuration example in Figure 51 is suitable when the resolution of the W image is to be improved, as in the second demosaicing example shown in Figure 7.

[0358] <18. Color correction processing in the color processing unit> Next, we will describe the color correction process performed by the color processing unit 18.

[0359] The color processing unit 18 performs color correction processing on the WRGB or WRGAB image data supplied from the demosaicing unit 17. The WRGB or WRGAB image data differs from the image data in the color space defined by the sRGB standard, which is a common display standard for displays. The color correction processing is the process of converting the WRGB or WRGAB image data into RGB3ch image data as defined by the sRGB standard.

[0360] Specifically, the color processing unit 18 performs a correction calculation using the linear matrix shown in equation (10) or equation (11) to determine the color space [R] defined in the sRGB standard. S ,G S ,B S Convert to RGB3ch image data. Equation (10) is the conversion formula when the pixel array is a WRGB array, and equation (11) is the conversion formula when the pixel array is a WRGAB array.

[0361]

number

[0362] When the pixel array is a WRGAB array, and pixel A is a pixel on which a color filter that transmits infrared (IR) light is placed, the imaging device 10 can capture images of both the visible light spectrum and the IR light spectrum without using mechanical switches, and can prevent any form of color corruption. Specifically, in the case of the spectral sensitivity characteristics shown in Figure 46, when shooting in daylight, all RGB visible light pixels react to infrared light, making it difficult to distinguish colors. At this time, in the matrix calculation of equation (11), k 04 , k 14 , k 24 By making it negative, color degradation can be prevented. On the other hand, in night photography, the k in equation (11) 04 , k 14 , k 24 By increasing this value, nighttime photography becomes possible.

[0363] When the pixel array is a WRGB array, the color filter formation process can be shortened by one step compared to a WRGAB array, resulting in lower costs and an increased bandwidth for the green (G) color component.

[0364] If the pixel array is a WRGAB array, and pixel A is a pixel with a color filter that transmits emerald green (EG), then color reproduction can be improved.

[0365] If the A pixels of the WRGAB array are assigned to colors of wavelengths other than RGB, specifically infrared (IR), cyan (C), emerald green (EG), orange (O), violet (V), or ultraviolet (UV), the imaging device 10 becomes a multispectral camera and can be used for high-resolution analysis of subjects.

[0366] <19. Example Data Format> Figure 52 shows an example of the RAW data output by the image sensor 12 and the image format after demosaicing and color correction processing.

[0367] Figure 52A shows the addresses of each pixel when the pixel array is a WRGAB array. The letters "W10", "W01", "W02", "W03", "R00", "G01", "R02", and "G03" shown in Figure 52A are the addresses of the pixels, which are added to indicate the correspondence with the data array.

[0368] Figure 52B shows the data format of the RAW data output by the image sensor 12. The data arrangement of the RAW data has the same number of data points horizontally as the number of horizontal pixels of the W pixels, and the same number of data points vertically as twice the number of vertical pixels of the W pixels. In the vertical direction, achromatic and chromatic pixel data are arranged alternately.

[0369] Figure 52C shows the data format of the image data after demosaicing by the demosaicing unit 17. The W image, R image, G image, B image, and A image have the same resolution, for example, the same resolution as the W pixel of the image sensor 12.

[0370] In Figure 52, section D shows the data format of the image data after color correction processing by the color processing unit 18. The color correction processing generates image data for the R image, G image, and B image. The R image, G image, and B image have the same resolution, for example, the same resolution as the W pixel of the image sensor 12.

[0371] Figures 53 and 54 show examples of RAW data output by the image sensor 12 and the image format after demosaicing and color correction processing, when the resolution of the W pixels is four times that of the chromatic pixels, as shown in the example in Figure 51. In the example in Figure 53, the shape of the W pixels is not octagonal for simplicity, but it may be octagonal as shown in the example in Figure 51. For convenience, one W pixel obtained by dividing the same pixel size as a chromatic pixel into four parts is referred to as a small W pixel.

[0372] Figure 53A shows the address of each pixel when the pixel array is a WRGAB array. The letters "W10", "W01", "W02", "W03", "R00", "G01", "R02", and "G03" shown in Figure 53A are the addresses of the pixels, which are added to indicate the correspondence with the data array.

[0373] Figure 53B shows the data format of the RAW data output by the image sensor 12. The data arrangement of the RAW data has the same number of data points horizontally as the number of horizontal chromatic pixels, and the number of data points vertically is five times the number of vertical chromatic pixels. In the vertical direction, data of four achromatic pixels and one chromatic pixel are arranged alternately, and the data of four consecutive achromatic pixels in the vertical direction corresponds to the data of four small W pixels corresponding to one chromatic pixel.

[0374] Figure 54A shows the data format of the image data after demosaicing by the demosaicing unit 17. The W image, R image, G image, B image, and A image have the same resolution, for example, the same resolution as the small W pixel of the image sensor 12.

[0375] Figure 54B shows the data format of the image data after color correction processing by the color processing unit 18. The color correction processing generates image data for the R image, G image, and B image. The R image, G image, and B image have the same resolution, for example, the same resolution as the small W pixel of the image sensor 12.

[0376] <20. Computer Configuration Examples> The image processing unit 24, which performs demosaicing by the demosaicing unit 17 and color correction by the color processing unit 18, may be provided separately from the imaging device 10 as an image processing device that executes these processes. The processing of the image processing unit 24 can be executed by hardware or by software. When a series of processes are executed by software, the programs that constitute the software are installed on a computer. Here, the computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0377] Figure 55 is a block diagram showing an example of the hardware configuration of a computer as an image processing device, in which the processing of the image processing unit 24 is executed by a program.

[0378] In a computer, the processing circuit 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected by a bus 904.

[0379] An input / output interface 905 is further connected to the bus 904. An input unit 906, an output unit 907, a storage unit 908, a communication unit 909, and a drive 910 are connected to the input / output interface 905.

[0380] The input unit 906 may include physical or virtual means of operation that the user operates to input information, such as a keyboard, mouse, or touch panel, as well as means of inputting information by the user through voice, gaze, etc. Furthermore, the input unit 906 may include sensors for inputting various physical quantities to the computer. For example, the input unit 906 may include sensors that acquire physical quantities such as light (including infrared light other than visible light) or sound, such as a camera or microphone. Also, for example, the input unit 906 may include sensors that acquire other physical quantities such as temperature, moisture content, acceleration, distance, etc. The output unit 907 may include means of presenting information to the user by stimulating the user's perception, such as a display, speaker, or haptic device. The storage unit 908 is composed of a hard disk, non-volatile or volatile memory, etc., and stores various types of information (including programs). The communication unit 909 is a network interface, etc., and performs wired or wireless communication with the outside. The drive 910 drives removable media 911 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0381] The processing circuit 901 includes a processor that executes programs such as a CPU (Central Processing Unit) and a DSP (Digital Signal Processor). The processing circuit 901 (its processor) performs the series of processes described above by loading the program stored in the memory unit 908 into the RAM 903 via the input / output interface 905 and the bus 904 and executing it. The processing circuit 901 can output the processing results of the series of processes from the output unit 907, for example, via the bus 904 and the input / output interface 905, as needed. The processing circuit 901 can also store the processing results in the memory unit 908 or transmit them from the communication unit 909.

[0382] The program executed by the computer (processing circuit 901) can be provided by recording it on a removable medium 911, such as a package medium. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

[0383] In a computer, a program can be installed in the storage unit 908 via the input / output interface 905 by inserting a removable media 911 into the drive 910. Alternatively, a program can be received by the communication unit 909 from another device, such as a server, via a wired or wireless transmission medium, and installed in the storage unit 908. Furthermore, programs can be pre-installed in the ROM 902 or the storage unit 908.

[0384] The programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

[0385] The processes that a computer performs according to a program do not necessarily have to follow the order described in the flowchart. In other words, the processes that a computer performs according to a program include processes that are executed in parallel or individually (e.g., parallel processing and object-based processing).

[0386] A program may be processed by a single computer (processor), or it may be processed in a distributed manner by multiple computers. Furthermore, a program may be transferred to a remote computer and executed there.

[0387] When the computer executes a program to perform the series of processes described above, the processing circuit 901 (its processor) functions as the demosaicing unit 17, the color processing unit 18, and so on.

[0388] The components (blocks) of the apparatus illustrated in this specification are functional conceptual blocks, and the actual apparatus does not need to have the illustrated configuration. That is, the apparatus can have any configuration in which the functions of the illustrated components are divided and / or integrated into any unit, for example, a configuration having one block in which the functions of all components are integrated.

[0389] The embodiments of this disclosure are not limited to those described above, and various modifications are possible without departing from the gist of the technology of this disclosure.

[0390] For example, the technology disclosed herein can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

[0391] Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices. In addition, if a single step includes multiple processes, those processes can be performed by a single device, or they can be divided and performed by multiple devices.

[0392] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0393] The technology disclosed herein can take the following configuration. (1) The image sensor has a demosaicing unit that performs demosaicing on an image generated by the image sensor, which has a first pixel array arranged in a diagonal grid pattern, the first pixel receiving light of a first color component consisting of red, green, and blue or yellow, the second pixel receiving light of a second color component consisting of red, the third pixel receiving light of a third color component consisting of green, and the fourth pixel receiving light of a fourth color component consisting of blue, or a second pixel array arranged in a diagonal grid pattern, the first pixel, the second pixel, the third pixel, the fourth pixel, and the fifth pixel receiving light of a fifth color component different from the first to fourth color components. The demosaicing section is, A first color component interpolation generation unit generates a first color component interpolation pixel value that interpolates the first color component for each pixel of the image other than the first pixel, A second color component virtual color difference generation unit generates a virtual color difference of the second color component based on the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, A third color component virtual color difference generation unit generates a virtual color difference of the third color component based on the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component, A fourth color component virtual color difference generation unit generates a virtual color difference of the fourth color component based on the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component, A fifth color component virtual color difference generation unit generates a virtual color difference of the fifth color component based on the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component, A first color component image generation unit generates a first color component image having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix using the pixel values ​​of the first pixels, and each pixel has the pixel value of the first color component. A second color difference component image generation unit generates a second color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel. A third color difference component image generation unit generates a third color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel. A fourth color difference component image generation unit generates a fourth color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel. A fifth color difference component image generation unit generates a fifth color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel. A first adder that generates a second color component image having the same arrangement and resolution as the first color component image by adding the second color difference component image and the first color component image, A second adder that generates a third color component image having the same arrangement and resolution as the first color component image by adding the third color difference component image and the first color component image, A third adder that generates a fourth color component image having the same arrangement and resolution as the first color component image by adding the fourth color difference component image and the first color component image, A fourth adder generates a fifth color component image having the same arrangement and resolution as the first color component image by adding the fifth color difference component image and the first color component image. has Image processing device. (2) The first color component image generation unit is: A pixel of interest setting unit sets each pixel of the first color component image to be generated as a pixel of interest, A filter tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as filter taps, A calculation unit generates the pixel value of the first color component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and a first tap coefficient that is stored in advance. has The image processing apparatus described in (1) above. (3) The first color component image generation unit is: A first code tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as a code tap. A first code calculation unit generates a first code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A first coefficient storage unit stores the first tap coefficient for each of the first codes and supplies the first tap coefficient corresponding to the first code generated by the first code calculation unit to the calculation unit. It further possesses, The calculation unit generates the pixel value of the first color component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and the first tap coefficient supplied from the first coefficient storage unit. The image processing apparatus described in (2) above. (4) The first tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the first color component image. The aforementioned student image is an image with the same resolution as the aforementioned image, and its pixel array is either the first pixel array or the second pixel array. The training image is an image in which the pixel array has the same array and resolution as the first color component image, and each pixel has the pixel value of the first color component. An image processing apparatus according to any of (2) to (3) above. (5) The second color difference component image generation unit is: A pixel of interest setting unit sets each pixel of the generated second color difference component image as a pixel of interest, A filter tap selection unit selects a plurality of virtual color difference pixels of the second color component surrounding the pixel of interest as filter taps, A calculation unit generates a pixel value of a second color difference component which is the color difference component between the first color component and the second color component of the pixel of interest by performing calculations using the pixel value of the selected filter tap and a second tap coefficient that is stored in advance. has An image processing apparatus according to any of (1) to (4) above. (6) The second color difference component image generation unit is: A color change code calculation unit that determines the color change code of the pixel of interest based on the dynamic range of the first and second pixels surrounding the pixel of interest, A second code tap selection unit selects a plurality of virtual color difference pixels of the second color component surrounding the pixel of interest as code taps, A second code calculation unit generates a second code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A second coefficient storage unit stores the second tap coefficient for each of the second codes and supplies the second tap coefficient corresponding to the second code generated by the second code calculation unit to the calculation unit. It further possesses, The calculation unit generates the pixel value of the second color difference component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and the second tap coefficient supplied from the second coefficient storage unit. The image processing apparatus described in (5) above. (7) The pixel value of the virtual color difference pixel of the second color component is calculated by the difference between the pixel value of the second pixel in the image and the first color component interpolation pixel value generated by the first color component interpolation generation unit. An image processing apparatus according to any of (5) to (6) above. (8) The second tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the second color difference component image. The aforementioned student image is an image with the same resolution as the aforementioned image, and its pixel array is either the first pixel array or the second pixel array. The training image is an image in which the pixel array has the same array and resolution as the second color component image, and each pixel has the pixel value of the second color component. An image processing apparatus according to any of (5) to (7) above. (9) The first color component interpolation generation unit is: A pixel of interest setting unit that sets each pixel other than the first pixel of the aforementioned image as a pixel of interest, A filter tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as filter taps, A calculation unit generates the first color component interpolated pixel value of the pixel of interest by performing calculations using the selected filter tap pixel value and a pre-stored third tap coefficient. has An image processing apparatus according to any of (1) to (8) above. (10) The first color component interpolation generation unit is: A third code tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as code taps, A third code calculation unit generates a third code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A third coefficient storage unit stores the third tap coefficient for each of the third codes and supplies the third tap coefficient corresponding to the third code generated by the third code calculation unit to the calculation unit. It further possesses, The calculation unit generates the first color component interpolated pixel value of the pixel of interest by performing calculations using the selected filter tap pixel value and the third tap coefficient supplied from the third coefficient storage unit. The image processing apparatus described in (9) above. (11) The third tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the aforementioned images. The aforementioned student image is an image in which the pixel array is the first pixel array or the second pixel array and has the same resolution as the aforementioned image, and the pixel values ​​of the first color component are located at the pixel positions of the first pixels. The training image is an image in which the pixel array is the first pixel array or the second pixel array and has the same resolution as the image, and the pixel values ​​of the first color component are located at pixel positions other than the first pixel. An image processing apparatus according to any of (9) to (10) above. (12) The color processing unit further comprises a color processing unit that generates RGB3ch image data from the first color component image, the second color component image, the third color component image, and the fourth color component image when the pixel array is the first pixel array, and generates RGB3ch image data from the first color component image, the second color component image, the third color component image, the fourth color component image, and the fifth color component image when the pixel array is the second pixel array. An image processing apparatus according to any of (1) to (11) above. (13) The image data generated by the image sensor has a format in which the number of data points in the horizontal direction is the same as the number of pixels in the horizontal direction of the first pixel, and the pixel data of the first pixel and the pixel data of pixels other than the first pixel are arranged alternately in the vertical direction. An image processing apparatus according to any of (1) to (12) above. (14) The aforementioned image sensor, The image processing apparatus described in (1) above and An imaging device equipped with the following features. (15) The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The nearest pixel 1 is composed of pixels with the same pixel size as the other pixels. The imaging device described in (14) above. (16) The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The first pixel is composed of an octagonal shape with a larger pixel size than the other pixels. The size of the on-chip lens provided for each pixel of the image sensor is the same for all pixels. An imaging device as described in any of (14) to (15) above. (17) The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The first pixel is composed of pixels obtained by dividing an octagonal shape, which is larger than the pixels other than the first pixel, into four parts. The size of the on-chip lens provided for each pixel of the image sensor is the same for the four octagonal first pixels and the pixels other than the first pixels. An imaging device as described in any of (14) to (16) above. (18) An optical low-pass filter is further provided between the imaging lens and the image sensor. The optical low-pass filter is composed of multiple birefringent optical elements, The birefringent optical element is configured such that the direction of separation of the incident light is the same as the direction of arrangement of the pixels arranged in the oblique grid. An imaging device as described in any of (14) to (17) above. (19) The image sensor has the second pixel array, The light of the fifth color component is infrared light. An imaging device as described in any of (14) to (18) above. (20) The process includes demosaicing an image generated by an image sensor having a first pixel array in which a first pixel receives light of a first color component consisting of red, green, and blue or yellow color components, a second pixel receives light of a second color component consisting of red, a third pixel receives light of a third color component consisting of green, and a fourth pixel receives light of a fourth color component consisting of blue, arranged in a diagonal grid pattern, or a second pixel array in which the first, second, third, and fourth pixels and a fifth pixel receives light of a fifth color component different from the first to fourth color components are arranged in a diagonal grid pattern. The aforementioned demosaicing process is For each pixel of the aforementioned image other than the first pixel, a first color component interpolation pixel value is generated to interpolate the first color component, With respect to the second pixel, a virtual color difference of the second color component is generated by the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, With respect to the third pixel, a virtual color difference of the third color component is generated by the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component, With respect to the fourth pixel, a virtual color difference of the fourth color component is generated by the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component, With respect to the fifth pixel, a virtual color difference of the fifth color component is generated by the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component. Using the pixel values ​​of the first pixels, a first-color component image is generated having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component. Using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel, a second color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel, a third color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel, a fourth color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel, a fifth color difference component image having the same arrangement and resolution as the first color component image is generated. By adding the second color difference component image and the first color component image, a second color component image having the same arrangement and resolution as the first color component image is generated. By adding the third color difference component image and the first color component image, a third color component image having the same arrangement and resolution as the first color component image is generated. By adding the fourth color difference component image and the first color component image, a fourth color component image having the same arrangement and resolution as the first color component image is generated. By adding the fifth color difference component image and the first color component image, a fifth color component image having the same arrangement and resolution as the first color component image is generated. including Image processing methods. (twenty one) On the computer, The process includes demosaicing of an image generated by an image sensor having a first pixel array in which a first pixel receives light of a first color component consisting of red, green, and blue or yellow color components, a second pixel receives light of a second color component consisting of red, a third pixel receives light of a third color component consisting of green, and a fourth pixel receives light of a fourth color component consisting of blue, arranged in a diagonal grid pattern, or a second pixel array in which the first, second, third, and fourth pixels and a fifth pixel receives light of a fifth color component different from the first to fourth color components are arranged in a diagonal grid pattern. The aforementioned demosaicing process is For each pixel of the aforementioned image other than the first pixel, a first color component interpolation pixel value is generated to interpolate the first color component, With respect to the second pixel, a virtual color difference of the second color component is generated by the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, With respect to the third pixel, a virtual color difference of the third color component is generated by the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component, With respect to the fourth pixel, a virtual color difference of the fourth color component is generated by the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component, With respect to the fifth pixel, a virtual color difference of the fifth color component is generated by the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component. Using the pixel values ​​of the first pixels, a first-color component image is generated having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component. Using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel, a second color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel, a third color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel, a fourth color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel, a fifth color difference component image having the same arrangement and resolution as the first color component image is generated. By adding the second color difference component image and the first color component image, a second color component image having the same arrangement and resolution as the first color component image is generated. By adding the third color difference component image and the first color component image, a third color component image having the same arrangement and resolution as the first color component image is generated. By adding the fourth color difference component image and the first color component image, a fourth color component image having the same arrangement and resolution as the first color component image is generated. By adding the fifth color difference component image and the first color component image, a fifth color component image having the same arrangement and resolution as the first color component image is generated. A program that performs a process that includes the following. [Explanation of symbols]

[0394] 10 Imaging device, 17 Demosaicing unit, 23 Virtual color difference (R-W') generation unit, 24 Virtual color difference (G-W') generation unit, 25 Virtual color difference (B-W') generation unit, 26 High-resolution W generation unit, 27 High-resolution (RW) generation unit, 28 High-resolution (GW) generation unit, 29 High-resolution (BW) generation unit, 30 Adder, 31 Adder, 32 Adder, 33 Virtual color difference (G-W') generation unit, 34 Virtual color difference (A-W') generation unit, 35 High-resolution (GW) generation unit, 36 High-resolution (AW) generation unit, 37 Adder, 38 Adder, 41 W' code tap selection unit, 42 W' filter tap selection unit, 43 W' code calculation unit, 44 W' coefficient storage unit, 45 W' sum-accumulate calculation unit, 46 Quantization calculation unit, 47 Conversion table storage unit, 51 W code tap selection unit, 52 W filter tap selection unit, 53 W code calculation unit, 54 W coefficient storage unit, 55 W sum-accumulate unit, 60 Color change amount code calculation unit, 61 (RW) code tap selection unit, 62 (RW) filter tap selection unit, 63 (RW) code calculation unit, 64 (RW) coefficient storage unit, 65 (RW) sum-accumulate unit, 100 learning device, 120 learning device, 130 learning device

Claims

1. The image sensor has a demosaicing unit that performs demosaicing on an image generated by the image sensor, which has a first pixel array in which the first pixel array is arranged in an oblique grid pattern, the first pixel receiving light of a first color component consisting of a first color component consisting of a red color component, the second pixel receiving light of a second color component consisting of a red color component, the third pixel receiving light of a third color component consisting of a green color component, and the fourth pixel receiving light of a fourth color component consisting of a blue color component. The demosaicing section is, A first color component interpolation generation unit generates a first color component interpolation pixel value that interpolates the first color component for each pixel other than the first pixel in the aforementioned image, A second color component virtual color difference generation unit generates a virtual color difference of the second color component based on the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, A third color component virtual color difference generation unit generates a virtual color difference of the third color component based on the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component, A fourth color component virtual color difference generation unit generates a virtual color difference of the fourth color component based on the difference between the pixel value of the fourth pixel and the first color component interpolated pixel value, A first color component image generation unit generates a first color component image having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix using the pixel values ​​of the first pixels, and each pixel has the pixel value of the first color component. A second color difference component image generation unit generates a second color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel. A third color difference component image generation unit generates a third color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel. A fourth color difference component image generation unit generates a fourth color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel. A first adder that generates a second color component image having the same arrangement and resolution as the first color component image by adding the second color difference component image and the first color component image, A second adder that generates a third color component image having the same arrangement and resolution as the first color component image by adding the third color difference component image and the first color component image, A third adder generates a fourth color component image having the same arrangement and resolution as the first color component image by adding the fourth color difference component image and the first color component image. has Image processing device.

2. The demosaicing unit can further perform demosaicing on an image generated by an image sensor having a second pixel array in which the first pixel, the second pixel, the third pixel, the fourth pixel, and the fifth pixel that receives light from the first color component to a fifth color component different from the fourth color component are arranged in an oblique grid. A fifth color component virtual color difference generation unit generates a virtual color difference of the fifth color component based on the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component, A fifth color difference component image generation unit generates a fifth color difference component image having the same arrangement and resolution as the first color component image, using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel. A fourth adder generates a fifth color component image having the same arrangement and resolution as the first color component image by adding the fifth color difference component image and the first color component image. It further possesses The image processing apparatus according to claim 1.

3. The first color component image generation unit is: A pixel of interest setting unit sets each pixel of the first color component image to be generated as a pixel of interest, A filter tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as filter taps, A calculation unit generates the pixel value of the first color component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and a first tap coefficient that is stored in advance. has The image processing apparatus according to claim 1.

4. The first color component image generation unit is: A first code tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as code taps. A first code calculation unit generates a first code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A first coefficient storage unit stores the first tap coefficient for each of the first codes and supplies the first tap coefficient corresponding to the first code generated by the first code calculation unit to the calculation unit. It further possesses, The calculation unit generates the pixel value of the first color component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and the first tap coefficient supplied from the first coefficient storage unit. The image processing apparatus according to claim 3.

5. The first tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the first color component image. The aforementioned student image is an image with a pixel array of the first pixel array and the same resolution as the aforementioned image. The training image is an image in which the pixel array has the same array and resolution as the first color component image, and each pixel has the pixel value of the first color component. The image processing apparatus according to claim 3.

6. The second color difference component image generation unit is: A pixel of interest setting unit sets each pixel of the generated second color difference component image as a pixel of interest, A filter tap selection unit selects a plurality of virtual color difference pixels of the second color component surrounding the pixel of interest as filter taps, A calculation unit generates a pixel value of a second color difference component which is the color difference component between the first color component and the second color component of the pixel of interest by performing calculations using the pixel value of the selected filter tap and a second tap coefficient that is stored in advance. has The image processing apparatus according to claim 1.

7. The second color difference component image generation unit is: A color change code calculation unit that determines the color change code of the pixel of interest based on the dynamic range of the first and second pixels surrounding the pixel of interest, A second code tap selection unit selects a plurality of virtual color difference pixels of the second color component surrounding the aforementioned pixel of interest as code taps, A second code calculation unit generates a second code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A second coefficient storage unit stores the second tap coefficient for each of the second codes and supplies the second tap coefficient corresponding to the second code generated by the second code calculation unit to the calculation unit. It further possesses, The calculation unit generates the pixel value of the second color difference component of the pixel of interest by performing calculations using the selected pixel value of the filter tap and the second tap coefficient supplied from the second coefficient storage unit. The image processing apparatus according to claim 6.

8. The pixel value of the virtual color difference pixel of the second color component is calculated by the difference between the pixel value of the second pixel in the image and the first color component interpolation pixel value generated by the first color component interpolation generation unit. The image processing apparatus according to claim 6.

9. The second tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the second color difference component image. The aforementioned student image is an image with a pixel array of the first pixel array and the same resolution as the aforementioned image. The training image is an image in which the pixel array has the same array and resolution as the second color component image, and each pixel has the pixel value of the second color component. The image processing apparatus according to claim 6.

10. The first color component interpolation generation unit is: A pixel of interest setting unit that sets each pixel other than the first pixel in the aforementioned image as a pixel of interest, A filter tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as filter taps, A calculation unit generates the first color component interpolation pixel value of the pixel of interest by performing calculations using the selected filter tap pixel value and a pre-stored third tap coefficient. has The image processing apparatus according to claim 1.

11. The first color component interpolation generation unit is: A third code tap selection unit that selects a plurality of first pixels surrounding the aforementioned pixel of interest as code taps, A third code calculation unit generates a third code representing the feature quantities of the surrounding pixels of the pixel of interest based on the pixel value of the selected code tap, A third coefficient storage unit stores the third tap coefficient for each of the third codes and supplies the third tap coefficient corresponding to the third code generated by the third code calculation unit to the calculation unit. It further possesses, The calculation unit generates the first color component interpolated pixel value of the pixel of interest by performing calculations using the selected filter tap pixel value and the third tap coefficient supplied from the third coefficient storage unit. The image processing apparatus according to claim 10.

12. The third tap coefficient is generated by learning using student images and teacher images generated from images with a higher resolution than the aforementioned images. The aforementioned student image is an image in which the pixel array is the first pixel array and has the same resolution as the aforementioned image, and the pixel values ​​of the first color component are located at the pixel positions of the first pixels. The training image is an image in which the pixel array is the first pixel array and has the same resolution as the image, and the pixel values ​​of the first color component are located at pixel positions other than the first pixel. The image processing apparatus according to claim 10.

13. The color processing unit further comprises a color processing unit that generates RGB 3ch image data from the first color component image, the second color component image, the third color component image, and the fourth color component image when the pixel array is the first pixel array, and generates RGB 3ch image data from the first color component image, the second color component image, the third color component image, the fourth color component image, and the fifth color component image when the pixel array is the second pixel array. The image processing apparatus according to claim 2.

14. The image data generated by the image sensor has a format in which the number of data points in the horizontal direction is the same as the number of pixels in the horizontal direction of the first pixel, and the pixel data of the first pixel and the pixel data of pixels other than the first pixel are arranged alternately in the vertical direction. The image processing apparatus according to claim 1.

15. The image processing apparatus according to claim 1, The image sensor having the first pixel array and An imaging device equipped with the following features.

16. The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The first pixel is composed of pixels with the same pixel size as the other pixels. The imaging apparatus according to claim 15.

17. The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The first pixel is composed of an octagonal shape with a larger pixel size than the other pixels. The size of the on-chip lens provided for each pixel of the image sensor is the same for all pixels. The imaging apparatus according to claim 15.

18. The image sensor has the first pixel and the pixels other than the first pixel arranged in a checkerboard pattern. The first pixel is composed of pixels obtained by dividing an octagonal shape, which is larger than the pixels other than the first pixel, into four parts. The size of the on-chip lens provided for each pixel of the image sensor is the same for the four octagonal first pixels and the pixels other than the first pixels. The imaging apparatus according to claim 15.

19. An optical low-pass filter is further provided between the imaging lens and the image sensor. The optical low-pass filter is composed of multiple birefringent optical elements, The birefringent optical element is configured such that the direction of separation of the incident light is the same as the direction of arrangement of the pixels arranged in the oblique grid. The imaging apparatus according to claim 15.

20. The image processing apparatus according to claim 2, The image sensor having the aforementioned two pixel arrays and Equipped with, The light of the fifth color component is infrared light. Imaging device.

21. The process includes demosaicing an image generated by an image sensor having a first pixel array in which a first pixel array is arranged in an oblique grid pattern, the first pixel receiving light of a first color component consisting of red, green, and blue or yellow color components, a second pixel receiving light of a second color component consisting of red, a third pixel receiving light of a third color component consisting of green, and a fourth pixel receiving light of a fourth color component consisting of blue. The aforementioned demosaicing process is For each pixel of the image other than the first pixel, a first color component interpolation pixel value is generated to interpolate the first color component, With respect to the second pixel, a virtual color difference of the second color component is generated by the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, With respect to the third pixel, a virtual color difference of the third color component is generated by the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component. With respect to the fourth pixel, a virtual color difference of the fourth color component is generated by the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component. Using the pixel values ​​of the first pixels, a first color component image is generated having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component. Using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel, a second color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel, a third color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel, a fourth color difference component image is generated that has the same arrangement and resolution as the first color component image. By adding the second color difference component image and the first color component image, a second color component image having the same arrangement and resolution as the first color component image is generated. By adding the third color difference component image and the first color component image, a third color component image having the same arrangement and resolution as the first color component image is generated. By adding the fourth color difference component image and the first color component image, a fourth color component image having the same arrangement and resolution as the first color component image is generated. including Image processing methods.

22. The image generated by the image sensor having a second pixel array in which the first pixel, the second pixel, the third pixel, the fourth pixel, and the fifth pixel that receives light from the first color component to the fourth color component that is different from the fifth color component are arranged in an oblique grid pattern can be further demosaiced. The aforementioned demosaicing process is With respect to the fifth pixel, a virtual color difference of the fifth color component is generated by the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component. Using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel, a fifth color difference component image having the same arrangement and resolution as the first color component image is generated. By adding the fifth color difference component image and the first color component image, a fifth color component image having the same arrangement and resolution as the first color component image is generated. Includes The image processing method according to claim 21.

23. On the computer, The process includes demosaicing of an image generated by an image sensor having a first pixel array in which the first pixel array is arranged in an oblique grid pattern, the first pixel receiving light of a first color component consisting of red, green, and blue or yellow color components, the second pixel receiving light of a second color component consisting of red, the third pixel receiving light of a third color component consisting of green, and the fourth pixel receiving light of a fourth color component consisting of blue. The aforementioned demosaicing process is For each pixel of the image other than the first pixel, a first color component interpolation pixel value is generated to interpolate the first color component, With respect to the second pixel, a virtual color difference of the second color component is generated by the difference between the pixel value of the second pixel and the interpolated pixel value of the first color component, With respect to the third pixel, a virtual color difference of the third color component is generated by the difference between the pixel value of the third pixel and the interpolated pixel value of the first color component. With respect to the fourth pixel, a virtual color difference of the fourth color component is generated by the difference between the pixel value of the fourth pixel and the interpolated pixel value of the first color component. Using the pixel values ​​of the first pixels, a first color component image is generated having a pixel array in which rectangular pixels parallel to the horizontal and vertical directions of the pixel array are arranged in a matrix, and each pixel has the pixel value of the first color component. Using the virtual color difference of the second color component, the pixel value of the first pixel, and the pixel value of the second pixel, a second color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the third color component, the pixel value of the first pixel, and the pixel value of the third pixel, a third color difference component image having the same arrangement and resolution as the first color component image is generated. Using the virtual color difference of the fourth color component, the pixel value of the first pixel, and the pixel value of the fourth pixel, a fourth color difference component image is generated that has the same arrangement and resolution as the first color component image. By adding the second color difference component image and the first color component image, a second color component image having the same arrangement and resolution as the first color component image is generated. By adding the third color difference component image and the first color component image, a third color component image having the same arrangement and resolution as the first color component image is generated. By adding the fourth color difference component image and the first color component image, a fourth color component image having the same arrangement and resolution as the first color component image is generated. including A program to execute a process.

24. The image generated by the image sensor having a second pixel array in which the first pixel, the second pixel, the third pixel, the fourth pixel, and the fifth pixel that receives light from the first color component to the fourth color component that is different from the fifth color component are arranged in an oblique grid pattern can be further demosaiced. The aforementioned demosaicing process is With respect to the fifth pixel, a virtual color difference of the fifth color component is generated by the difference between the pixel value of the fifth pixel and the interpolated pixel value of the first color component. Using the virtual color difference of the fifth color component, the pixel value of the first pixel, and the pixel value of the fifth pixel, a fifth color difference component image having the same arrangement and resolution as the first color component image is generated. By adding the fifth color difference component image and the first color component image, a fifth color component image having the same arrangement and resolution as the first color component image is generated. Includes The program according to claim 23.

Images