Image processing system, image processing device, image processing method, and program

Abstract

The present invention improves image quality in an image processing system for processing a color image captured using a polarization filter.　This image processing system includes a false color amount calculation unit and a noise reduction strength calculation unit. In the image processing system, the false color amount calculation unit calculates a false color amount after polarization signal processing, which is polarization component-related processing. On the basis of the false color amount, the noise reduction strength calculation unit calculates, as a noise reduction strength, the degree of noise reduction processing for a color image captured using a polarization filter.

Description

Image processing system, image processing device, image processing method, and program

[0001] The present technology relates to an image processing system, and more particularly to an image processing system, an image processing device, an image processing method, and a program for processing color images.

[0002] Conventionally, various image processing techniques such as noise reduction have been used in image processing devices and image processing systems. For example, a device has been proposed that calculates the degree of false color that occurs in a color image after HDR (High-Dynamic-Range) composition as a false color intensity value and performs false color correction processing according to the false color intensity value (see, for example, Patent Document 1).

[0003] JP 2015-33107 A

[0004] In the above-described conventional technology, false colors caused by HDR compositing are corrected by performing false color correction processing. However, false colors can also occur due to causes other than HDR compositing. For example, if a color image is captured using a polarizing filter, false colors may occur due to white balance correction or development processing, and the false colors may degrade image quality. While false colors caused by white balance correction or the like can be suppressed by further performing noise reduction processing and increasing its intensity, excessive noise reduction processing is not preferable because it may actually degrade image quality.

[0005] This technology was developed in light of these circumstances, and aims to improve image quality in an image processing system that processes color images captured using a polarizing filter.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is an image processing system including a false color amount calculation unit that calculates the amount of false color after polarization signal processing, which is processing related to polarization components, and a noise reduction intensity calculation unit that calculates the degree of noise reduction processing for a color image captured using a polarization filter as noise reduction intensity based on the false color amount, an image processing method for the system, and a program for causing a computer to execute the method, thereby improving image quality.

[0007] In addition, in the first aspect, the image processing apparatus may further include a noise estimation processing unit that calculates a noise amount after white balance correction as a noise estimation amount and outputs the calculated amount to the false-color amount calculation unit, a noise reduction processing unit that performs the noise reduction processing at the noise reduction intensity on the color image and outputs the resulting noise-reduced image, a white balance correction unit that performs the white balance correction on the noise-reduced image and outputs the resulting white-balance-corrected image, a development processing unit that performs a development processing on the white balance-corrected image and outputs the resulting output image, and a polarization signal processing unit that performs the polarization signal processing on the output image, and the false-color amount calculation unit calculates the false-color amount from the noise estimation amount, thereby achieving the effect of calculating the false-color amount from the estimated amount of noise.

[0008] In addition, in this first aspect, the color image may include levels of OPB pixels, and the noise estimation processing unit may calculate the noise estimation amount based on statistics of the levels of the OPB pixels and a white balance gain used in the white balance correction, thereby providing an effect that the amount of noise is estimated from the levels of the OPB pixels.

[0009] In addition, in this first aspect, the noise estimation processing unit may calculate the noise estimation amount based on a temperature measured in the image processing system, at least one of an analog gain, a digital gain, and an exposure time used in the image processing system, and a white balance gain used in the white balance correction, thereby eliminating the need for OPB pixels.

[0010] In addition, in the first aspect, the image processing apparatus may further include a white balance correction unit that performs white balance correction on the color image and outputs the result as a white balance corrected image, a noise detection processing unit that detects an amount of noise after the white balance correction and outputs the result to a false color amount calculation processing unit, a development processing unit that performs the noise reduction processing of the white balance corrected image using the noise reduction intensity together with a development processing and outputs the result as an output image, and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the amount of false color from the detected amount of noise, thereby providing the effect of calculating the amount of false color from the detected amount of noise.

[0011] In addition, in the first aspect, the imaging device may further include a white balance correction unit that performs white balance correction on the color image and outputs the result as a white balance corrected image, a development processing unit that performs development processing on the white balance corrected image and outputs the result as an output image, a noise detection processing unit that detects an amount of noise after development processing and outputs the detected amount of noise to a false color amount calculation processing unit, and a polarization signal processing unit that performs the polarization signal processing and the noise reduction processing of the noise reduction intensity on the output image, wherein the false color amount calculation unit calculates the amount of false color from the detected amount of noise. This brings about an effect of reducing the amount of processing of the imaging device that performs development processing.

[0012] In addition, in the first aspect, the image processing apparatus may further include a white balance correction unit that performs white balance correction on the color image and outputs the resultant image as a white balance corrected image, a development processing unit that performs development processing on the white balance corrected image and outputs the resultant image as an output image, and a polarization signal processing unit that performs the polarization signal processing on the output image and outputs the resultant image as a processed image, and the false color amount calculation unit may calculate the amount of false color generated in the processed image, thereby eliminating the need to estimate the amount of noise.

[0013] In addition, in the first aspect, the image processing device may further include a noise reduction processing unit that performs the noise reduction processing with the noise reduction intensity on the color image and outputs the result to the white balance correction unit, thereby providing an effect of feeding back the noise reduction intensity to the noise reduction processing unit.

[0014] In this first aspect, the development processing unit may further perform the noise reduction processing of the noise reduction intensity together with the development processing, thereby providing an effect of feeding back the noise reduction intensity to the development processing unit.

[0015] In addition, in this first aspect, the polarization signal processing unit may further perform the noise reduction processing of the noise reduction intensity in addition to the polarization signal processing, thereby providing an effect of feeding back the noise reduction intensity to the polarization signal processing unit.

[0016] According to a second aspect of the present technology, there is provided an image processing device including: a false color amount calculation unit that calculates a false color amount after polarization signal processing, which is processing related to polarization components; and a noise reduction strength calculation unit that calculates a degree of noise reduction processing for a color image captured using a polarization filter based on the false color amount, thereby improving image quality.

[0017] In addition, in the second aspect, the image processing apparatus may further include a noise estimation processing unit that calculates a noise amount after white balance correction as a noise estimation amount and outputs the calculated amount of noise to the false-color amount calculation unit, a noise reduction processing unit that performs the noise reduction processing at the noise reduction intensity on the color image and outputs the resulting noise-reduced image, a white balance correction unit that performs the white balance correction on the noise-reduced image and outputs the resulting white-balance-corrected image, a development processing unit that performs a development processing on the white balance-corrected image and outputs the resulting output image, and a polarization signal processing unit that performs the polarization signal processing on the output image, and the false-color amount calculation unit calculates the false-color amount from the noise estimation amount, thereby achieving the effect of calculating the false-color amount from the estimated amount of noise.

[0018] In addition, in this second aspect, the color image may include levels of OPB pixels, and the noise estimation processing unit may calculate the noise estimation amount based on statistics of the OPB pixel levels and a white balance gain used in the white balance correction, thereby providing an effect that the noise amount is estimated from the OPB pixel levels.

[0019] In addition, in this second aspect, the noise estimation processing unit may calculate the noise estimation amount based on a temperature measured in the image processing device, at least one of an analog gain, a digital gain, and an exposure time used in the image processing device, and a white balance gain used in the white balance correction, thereby eliminating the need for OPB pixels.

[0020] In addition, in the second aspect, the image processing apparatus may further include a white balance correction unit that performs the white balance correction on the color image and outputs the result as a white balance corrected image, a noise detection processing unit that detects an amount of noise after the white balance correction and outputs the result to a false color amount calculation processing unit, a development processing unit that performs the noise reduction processing on the white balance corrected image using the noise reduction intensity together with a development processing and outputs the result as an output image, and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the amount of false color from the detected amount of noise, thereby providing the effect of calculating the amount of false color from the detected amount of noise.

[0021] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example of a configuration of an imaging device according to the first embodiment of the present technology. FIG. 3 is a block diagram showing an example of a configuration of an image sensor according to the first embodiment of the present technology. FIG. 4 is a diagram showing an example of a configuration of a pixel array unit according to the first embodiment of the present technology. FIG. 5 is a diagram showing an example of a structure of a pixel block according to the first embodiment of the present technology. FIG. 6 is a diagram showing an example of a control method when a polarizing plate is provided according to the first embodiment of the present technology. FIG. 7 is a block diagram showing an example of a configuration of an image processing unit and an information processing device according to the first embodiment of the present technology. FIG. 8 is a graph showing an example of a relationship between a luminance value and a polarization angle according to the first embodiment of the present technology. FIG. 9 is a graph showing an example of a relationship between a luminance value and a polarization angle before noise reduction processing in a comparative example. FIG. 10 is a graph showing an example of a relationship between a luminance value and a polarization angle after white balance correction in a comparative example. FIG. 11 is a graph showing an example of a relationship between a luminance value and a polarization angle after polarization signal processing in a comparative example. FIG. 12 is a diagram showing an example of setting an ROI (Region Of Interest) and an example of screen division according to the first embodiment of the present technology. FIG. 13 is a graph showing an example of a relationship between a false color estimation amount, a noise estimation amount, and NR (Noise Reduction) strength according to the first embodiment of the present technology. 1 is a flowchart showing an example of operation of an image processing system according to a first embodiment of the present technology. FIG. 2 is a block diagram showing an example configuration of an image sensor according to a modified example of the first embodiment of the present technology. FIG. 3 is a diagram for explaining a method for estimating a noise amount according to a modified example of the first embodiment of the present technology. FIG. 4 is a block diagram showing an example configuration of an image processing unit and an information processing device according to a second embodiment of the present technology. FIG. 5 is a flowchart showing an example of operation of an image processing system according to the second embodiment of the present technology. FIG. 6 is a block diagram showing an example configuration of an image processing unit and an information processing device according to a third embodiment of the present technology. FIG. 7 is a flowchart showing an example of operation of an image processing system according to the third embodiment of the present technology. FIG. 8 is a block diagram showing an example configuration of an image processing unit and an information processing device according to a fourth embodiment of the present technology. FIG. 9 is a flowchart showing an example of operation of an image processing system according to the fourth embodiment of the present technology.13 is a block diagram showing an example of a configuration of an image processing unit and an information processing device according to a first modified example of the fourth embodiment of the present technology;FIG. 14 is a block diagram showing an example of a configuration of an image processing unit and an information processing device according to a second modified example of the fourth embodiment of the present technology;FIG.

[0022] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be given in the following order: 1. First embodiment (an example in which the amount of false color and noise reduction strength are calculated) 2. Second embodiment (an example in which the amount of noise is detected before development and the amount of false color and noise reduction strength are calculated) 3. Third embodiment (an example in which the amount of noise is detected after development and the amount of false color and noise reduction strength are calculated) 4. Fourth embodiment (an example in which the amount of false color and noise reduction strength are calculated and fed back)

[0023] 1 is a block diagram showing an example of the configuration of an image processing system 100 according to a first embodiment of the present technology. The image processing system 100 is a system for processing color images, and includes an imaging device 110 and an information processing device 120.

[0024] The imaging device 110 is a camera equipped with an imaging element having multiple polarizers, and captures color images. This imaging device 110 is also called a polarization camera, and is used, for example, as an industrial camera. The imaging device 110 performs processing such as development on the captured color images, and transmits the images to the information processing device 120 via wired or wireless communication. The imaging device 110 is an example of an image processing device as defined in the claims.

[0025] The information processing device 120 performs polarization signal processing, which is signal processing related to polarization components. This polarization signal processing includes, for example, processing to extract polarization components and processing to suppress reflection. The information processing device 120 may be, for example, a personal computer.

[0026] 2 is a block diagram showing an example of the configuration of the imaging device 110 according to the first embodiment of the present technology. The imaging device 110 includes an optical unit 111, an image sensor 200, and a DSP (Digital Signal Processing) circuit 112. The imaging device 110 further includes an operation unit 113, a bus 114, a frame memory 115, a storage unit 116, a power supply unit 117, and a communication unit 118.

[0027] The optical unit 111 collects light from a subject and guides it to the image sensor 200. The image sensor 200 generates color image data by photoelectric conversion. The image sensor 200 outputs the generated image data to the DSP circuit 112.

[0028] The DSP circuit 112 performs predetermined signal processing on the image data and outputs the processed image data to a frame memory 115 and a communication unit 118 via a bus 114.

[0029] The operation unit 113 generates an operation signal in accordance with a user's operation.

[0030] The bus 114 is a common path for the optical unit 111, image sensor 200, DSP circuit 112, operation unit 113, frame memory 115, storage unit 116, power supply unit 117, and communication unit 118 to exchange data with one another.

[0031] The frame memory 115 holds image data. The storage unit 116 stores various data such as image data. The power supply unit 117 supplies power to the image sensor 200, the DSP circuit 112, and the like.

[0032] The communication unit 118 transmits image data to the information processing device 120 .

[0033] 3 is a block diagram showing an example of the configuration of the image sensor 200 according to the first embodiment of the present technology. The image sensor 200 includes a vertical drive circuit 211, a timing control unit 212, a DAC (Digital to Analog Converter) 213, a pixel array unit 220, a column signal processing unit 214, a horizontal transfer control unit 215, and an image processing unit 250.

[0034] A plurality of pixels are arranged in a two-dimensional lattice pattern in the pixel array section 220. The vertical drive circuit 211 sequentially selects and drives rows in the pixel array section 220.

[0035] The timing control section 212 controls the operation timing of the vertical drive circuit 211, the DAC 213, the column signal processing section 214, and the horizontal transfer control section 215 in synchronization with the vertical synchronization signal XVS.

[0036] The DAC 213 generates a sawtooth ramp signal and outputs it to the column signal processing unit 214 .

[0037] The column signal processing unit 214 performs processes such as AD (Analog to Digital) conversion and CDS (Correlated Double Sampling) on ​​pixel signals for each column. The column signal processing unit 214 outputs a color image in which the processed pixel signals are arranged as an input image to the image processing unit 250.

[0038] The horizontal transfer control unit 215 controls the column signal processing unit 214 to output pixel signals in sequence.

[0039] The image processing unit 250 performs various image processing such as white balance correction and development processing on the input image from the column signal processing unit 214. The image processing unit 250 outputs the processed image to the DSP circuit 112 as an output image.

[0040] 4 is a diagram showing an example of the configuration of the pixel array unit 220 according to the first embodiment of the present technology. The pixel array unit 220 is divided into a VOPB region 221, a HOPB region 222, and a light receiving region 223.

[0041] The VOPB region 221 is light-shielded, and a plurality of OPB (optical black) pixels 230 are arranged in each column in the column direction. The HOPB region 222 is light-shielded, and a plurality of OPB pixels 230 are arranged in each row in the row direction.

[0042] The light receiving area 223 is not light-shielded and has an array of multiple light receiving pixels. The light receiving pixels include, for example, R (Red) pixels, G (Green) pixels, and B (Blue) pixels. The R pixels photoelectrically convert incident red light to generate pixel signals, the G pixels photoelectrically convert incident green light to generate pixel signals, and the B pixels photoelectrically convert incident blue light to generate pixel signals.

[0043] The light receiving area 223 is divided into a predetermined number of pixel blocks 241, a predetermined number of pixel blocks 242, and a predetermined number of pixel blocks 243. Four R pixels are arranged in two rows and two columns in the pixel block 241. Four G pixels are arranged in two rows and two columns in the pixel block 242. Four B pixels are arranged in two rows and two columns in the pixel block 243.

[0044] Focusing on an area in which four pixel blocks are arranged in two rows and two columns, two pixel blocks 242 are arranged diagonally, and pixel blocks 241 and 242 are arranged in the remaining positions. This type of arrangement is called a Quad Bayer arrangement.

[0045] 5 is a diagram showing an example of the structure of a pixel block 241 according to the first embodiment of the present technology. The pixel block 241 includes on-chip lenses 311 to 314, a color filter 320, a wire grid polarizer 330, and photodiodes 341 to 344. In the drawing, the side closer to the optical unit 111 (not shown) is defined as the "up" direction.

[0046] The photodiodes 341 to 344 are arranged in two rows and two columns. These photodiodes generate electric charges through photoelectric conversion. A circuit that accumulates the electric charges and generates pixel signals according to the amount of electric charge is formed below the photodiodes 341 to 344.

[0047] A wire grid polarizer 330 is disposed above the photodiode. This wire grid polarizer 330 includes polarizing filters 331 to 334 arranged in a 2-row by 2-column array. Each of the polarizing filters 331 to 334 transmits light polarized in a different direction. For example, if the polarization angle of light transmitted through polarizing filter 331 is 0 degrees, the polarization angles corresponding to polarizing filters 332, 333, and 334 are 45 degrees, 90 degrees, and 135 degrees, respectively.

[0048] A color filter 320 that transmits red light is disposed on top of the wire grid polarizer 330 .

[0049] Above the color filter 320, on-chip lenses 311 to 314 are arranged in two rows and two columns.

[0050] The pixel blocks 242 and 243 have the same configuration as the pixel block 241, except that the colors transmitted by the color filters are different.

[0051] As shown in the figure, by arranging a polarizing filter 331 or the like for each pixel, each light-receiving pixel (R pixel, G pixel, or B pixel) can receive incident light that has passed through that polarizing filter. Furthermore, by using a wire grid polarizer 330 that includes four polarizing filters with different polarization angles, each light-receiving pixel can simultaneously receive four polarized light components with different polarization angles for each of the R, G, and B colors.

[0052] Although the image sensor 200 generates a color image using the wire grid polarizer 330 , a similar color image can also be generated without using the wire grid polarizer 330 .

[0053] When the wire grid polarizer 330 is not used, a polarizing plate 270 that covers all the pixels is instead disposed on top of the pixel array section 220. The polarization angle of the polarizing plate 270 is variable, and the polarization angle is controlled by the timing control section 212 or the like.

[0054] 6 , the image sensor 200 switches the polarization angle of the polarizing plate 270 between 0 degrees, 45 degrees, 90 degrees, and 135 degrees in sequence, and captures an image for each polarization angle. This allows the image sensor 200 to generate a color image similar to that in the case where the wire grid polarizer 330 is used. When the polarizing plate 270 is used, the pixel array unit 220 arranged in a normal Bayer array can be used instead of the quad Bayer array.

[0055] [Configuration example of image processing unit] Fig. 7 is a block diagram showing a configuration example of the image processing unit 250 and the information processing device 120 according to the first embodiment of the present technology.

[0056] The image processing unit 250 includes a noise estimation processing unit 251, a false color estimation processing unit 252, an NR (Noise Reduction) intensity calculation unit 253, and a noise reduction processing unit 254. The image processing unit 250 further includes a shading correction unit 255, a digital gain processing unit 256, a white balance correction unit 257, and a development processing unit 258.

[0057] A color image in which the luminance values ​​from each of the R, G, and B pixels and the levels of the OPB pixels 230 are arranged is input to the image processing unit 250 as an input image. This input image is an image before development processing, and is also called a mosaic image or a RAW image. The R, G, and B luminance values ​​are input to a false color estimation processing unit 252 and a noise reduction processing unit 254. Meanwhile, the levels of the OPB pixels 230 are input to a noise estimation processing unit 251.

[0058] The noise estimation processing unit 251 estimates the amount of noise after subsequent white balance correction. This noise estimation processing unit 251 calculates the average values ​​of the levels of the HOPB area 222 and the VOPB area 221 for each row and for each column. Then, for each pixel, the noise estimation processing unit 251 calculates the statistics (average or sum) of the average values ​​of the corresponding row and the average values ​​of the corresponding column as the noise amount N of that pixel. ｃａｌｃ The noise amount N for each of the R, G, and B pixels is calculated as follows: ｃａｌｃ N ｃａｌｃ＿Ｒ , N ｃａｌｃ＿Ｇ , N ｃａｌｃ＿Ｂ Let's say.

[0059] Next, the noise estimation processing unit 251 calculates a digital gain DG_R for the luminance value of the R pixel, a white balance gain WBG_R for the luminance value of the R pixel, and a noise amount N ｃａｌｃ＿Ｒ From this, the noise estimate N of the R pixel is calculated using the following formula: ｅｓｔ＿Ｒ Calculate N ｅｓｔ＿Ｒ = N ｃａｌｃ＿Ｒ ×DG_R×WBG_R ... Formula 1

[0060] The noise estimation processing unit 251 calculates the noise estimation amount N of the B pixel by the following formula: ｅｓｔ＿Ｂ Calculate N ｅｓｔ＿Ｂ = N ｃａｌｃ＿Ｂ ×DG_B×WBG_B ... Formula 2

[0061] The noise estimation processing unit 251 calculates the noise estimation amount N ｅｓｔ＿Ｒ and N ｅｓｔ＿Ｂ to the false color estimation processing unit 252.

[0062] The false color estimation processing unit 252 estimates the amount of false color after the subsequent polarization signal processing. The cause of false color will be described later.

[0063] Here, the luminance values ​​of the R pixels at polarization angles of 0, 45, 90, and 135 degrees are expressed as I ０＿Ｒ , I ４５＿Ｒ , I ９０＿Ｒ , I １３５＿Ｒ The luminance values ​​of the G pixels at polarization angles of 0, 45, 90, and 135 degrees are expressed as I ０＿Ｇ , I ４５＿Ｇ , I ９０＿Ｇ , I １３５＿Ｇ The luminance values ​​of the B pixels at polarization angles of 0, 45, 90, and 135 degrees are expressed as I ０＿Ｂ , I ４５＿Ｂ , I ９０＿Ｂ , I １３５＿Ｂ Let's say.

[0064] The false color estimation processing unit 252 ０＿Ｒ , I ４５＿Ｒ , I ９０＿Ｒ , I １３５＿Ｒ Using the following formula, coefficient A Ｒ , B Ｒ , C Ｒ Calculate the following. Ｒ = (I４５＿Ｒ -I １３５＿Ｒ ) / 2...Formula 3 B Ｒ = (I ０＿Ｒ -I ９０＿Ｒ ) / 2...Formula 4 C Ｒ = (I ０＿Ｒ +I ４５＿Ｒ +I ９０＿Ｒ +I １３５＿Ｒ ) / 4 ...Equation 5

[0065] Then, the false color estimation processing unit 252 calculates the coefficient A Ｒ and B Ｒ From the above, the polarization component ρ of the R pixel is calculated by the following formula: Ｒ Calculate ρ Ｒ = (A Ｒ ２ +B Ｒ ２ ) １／２ ...Formula 6

[0066] Furthermore, the false color estimation processing unit 252 calculates the coefficient C of the G pixel by replacing R in Equations 3 to 6 with G. Ｇ and polarization component ρ Ｇ Calculate the coefficient C of the B pixel by replacing R in Equations 3 to 6 with B. Ｂ and polarization component ρ Ｂ Calculate.

[0067] Next, the false color estimation processing unit 252 generates an image in which C-ρ is arranged as a reflection-suppressed image. Ｒ , C Ｇ or C Ｂ and ρ is ρ Ｒ , ρ Ｇ or ρ Ｂ is.

[0068] The false color estimation processing unit 252 then selects an area in the reflection-suppressed image for which to calculate the amount of false color. For example, it is assumed that the image processing unit 250 has set a predetermined number of ROIs in the reflection-suppressed image by image recognition or the like. In this case, the false color estimation processing unit 252 calculates, for each ROI, statistics (total or average) of the luminance values ​​within the ROI. The false color estimation processing unit 252 selects ROIs for which the luminance value statistics is equal to or less than a threshold value as calculation targets, and excludes ROIs for which the luminance value statistics exceeds the threshold value from the calculation targets.

[0069] Alternatively, the false color estimation processor 252 divides the reflection-suppressed image into multiple areas, calculates the luminance statistics (total or average) for each area, and selects areas where the luminance statistics is equal to or less than a threshold as calculation targets, and excludes areas where the luminance statistics exceeds the threshold from the calculation targets.

[0070] Alternatively, the false color estimation processing unit 252 may perform calculations on the entire image, rather than on a portion of the image.

[0071] Furthermore, the false color estimation processing unit 252 calculates I ０＿Ｒ , I ４５＿Ｒ , I ９０＿Ｒ , I １３５＿Ｒ and the noise estimator N ｅｓｔ＿Ｒ From this, I is calculated by the following formula: ０＿Ｒ ', I ４５＿Ｒ ', I ９０＿Ｒ ', I １３５＿Ｒ ' is calculated. ０＿Ｒ '=I ０＿Ｒ +N ｅｓｔ＿Ｒ ...Equation 7I ４５＿Ｒ '=I ４５＿Ｒ +N ｅｓｔ＿Ｒ ...Formula 8 I ９０＿Ｒ '=I ９０＿Ｒ +N ｅｓｔ＿Ｒ ...Equation 9 I １３５＿Ｒ '=I １３５＿Ｒ +N ｅｓｔ＿Ｒ ...Formula 10

[0072] Then, the false color estimation processing unit 252 calculates the estimated polarization component ρ of the R pixel in the calculation target area using the following formula: Ｒ ' is calculated.Ｒ '=(I ４５＿Ｒ '-I １３５＿Ｒ ') / 2...Formula 11 B Ｒ '=(I ０＿Ｒ '-I ９０＿Ｒ ') / 2...Equation 12 ρ Ｒ '=(A Ｒ ' ２ +B Ｒ ' ２ ) １／２ ...Formula 13

[0073] Furthermore, the false color estimation processing unit 252 calculates the estimated polarization component ρ of the G pixel by replacing R in Equations 7 to 13 with G. Ｒ ' is calculated, and the estimated polarization component ρ of the B pixel is calculated by replacing R in Equations 7 to 13 with B. Ｂ ' is calculated.

[0074] Then, the false color estimation processing unit 252 calculates the false color amount FC of the R pixel in the calculation target area using the following formula: ｅｓｔ＿Ｒ Calculate ρ ｅｓｔ＿Ｒ = (ρ Ｒ '-ρ Ｒ ) / (ρ Ｇ '-ρ Ｇ )...Formula 14 FC ｅｓｔ＿Ｒ = ρ ｅｓｔ＿Ｒ / (C Ｇ -ρ Ｇ )...Equation 15

[0075] Furthermore, the false color estimation processing unit 252 calculates the false color amount FC of the B pixel by replacing R in Equation 14 and Equation 15 with B. ｅｓｔ＿Ｂ Calculate.

[0076] The false color estimation processing unit 252 calculates the false color amount FC ｅｓｔ＿Ｒ and F.C. ｅｓｔ＿Ｂ is output to the NR intensity calculation unit 253.

[0077] The false color estimation processing unit 252 is an example of a false color amount calculation unit described in the claims.

[0078] The NR intensity calculation unit 253 calculates the false color amount FC ｅｓｔ＿Ｒ and F.C. ｅｓｔ＿ＢThe NR intensity calculation unit 253 calculates the degree of noise reduction processing for the input image as the NR intensity based on the false color amount FC ｅｓｔ＿Ｒ The larger the value, the higher the NR intensity Deg of the R pixel. Ｒ is calculated, and the false color amount FC ｅｓｔ＿Ｂ The larger the value, the higher the NR intensity Deg of the B pixel. Ｂ Calculate.

[0079] The NR intensity calculation unit 253 calculates the false color amount FC ｅｓｔ＿Ｒ and F.C. ｅｓｔ＿Ｂ In addition to the noise estimator N ｅｓｔ＿Ｒ and N ｅｓｔ＿Ｂ In this case, the NR intensity calculation unit 253 calculates the false color amount FC ｅｓｔ＿Ｒ The larger the value, the higher the NR intensity Deg of the R pixel. Ｒ＿１ and the noise estimator N ｅｓｔ＿Ｒ The larger the value of NR intensity Deg Ｒ＿２ and their statistics (average, etc.) are used as the NR intensity Deg. Ｒ By the same calculation, the NR intensity Deg of the B pixel is calculated as follows: Ｂ is also calculated.

[0080] The NR strength calculation unit 253 calculates the NR strength Deg Ｒ and Deg. Ｂ is output to the noise reduction processing unit 254.

[0081] The noise reduction processing unit 254 performs NR intensity Deg on the input image. Ｒ and Deg. Ｂ The noise reduction process is performed by performing two-dimensional noise reduction (NR) in the spatial direction and three-dimensional noise reduction (NR) in the time axis direction. For example, edge-preserving NR or band-splitting NR is used as the two-dimensional NR. For edge-preserving NR, an epsilon filter or a bilateral filter is used.

[0082] For example, when an epsilon filter is used, the correction amount E of the pixel of interest is calculated by the following formula.

[0083] In the above formula, n is the number of surrounding pixels surrounding the pixel of interest. ｋ is a weighting coefficient corresponding to the kth (k is an integer from 1 to n) peripheral pixel, and the value of the NR intensity calculated by the NR intensity calculation unit 253 is set. ｋ is the difference between the pixel of interest and the k-th surrounding pixel.

[0084] The noise reduction processing unit 254 outputs the processed image to the shading correction unit 255 as an NR image.

[0085] The shading correction unit 255 performs shading correction on the NR image and outputs the corrected image to the digital gain processing unit 256 as a corrected image.

[0086] The digital gain processing unit 256 amplifies the luminance value of each pixel in the corrected image by the digital gain DG. The luminance values ​​of R pixels and B pixels are amplified by the digital gains DG_R and DG_B, respectively, while the luminance value of G pixels is not amplified. The digital gain processing unit 256 outputs the amplified image to the white balance correction unit 257 as an amplified image.

[0087] Although both the shading correction unit 255 and the digital gain processing unit 256 are provided, one or both of them may not be provided.

[0088] The white balance correction unit 257 amplifies the luminance value of each pixel in the amplified image by the white balance gain WBG. In this white balance correction, the luminance values ​​of the R and B pixels are amplified (in other words, corrected) by the white balance gains WBG_R and WBG_B, respectively, while the luminance value of the G pixel is not corrected. The white balance correction unit 257 outputs the corrected image to the development processing unit 258 as a WB-corrected image.

[0089] The development processing unit 258 performs development processing on the WB corrected image (in other words, the mosaic image) to interpolate missing color information for each pixel. This development processing is also called demosaic processing. The development processing unit 258 transmits the processed image as an output image to the information processing device 120 via the DSP circuit 112 (not shown) or the like.

[0090] The information processing device 120 includes a polarization signal processing unit 121. The polarization signal processing unit 121 performs polarization signal processing, which is processing related to polarization components, on the output image. The polarization signal processing unit 121 outputs the processed image to an external monitor or the like as a processed image.

[0091] For example, polarization signal processing may involve generating an average image by arranging the coefficients C of each pixel, or generating a polarization component image by arranging the polarization components ρ of each pixel.

[0092] Alternatively, a process for generating an image in which C / ρ of each pixel is arranged as a polarization degree image, or a process for generating the above-mentioned reflection suppressed image, is executed as polarization signal processing.

[0093] Alternatively, polarization signal processing may involve generating an image in which C+ρ of each pixel is arranged as a reflection coordinated image, or processing a polarization degree image to generate a polarization direction image in which the polarization direction is represented by color coding.

[0094] Next, the cause of false color will be described with reference to FIGS.

[0095] 8 is a graph showing an example of the relationship between luminance value and polarization angle in the first embodiment of the present technology. The vertical axis in the figure represents luminance value, and the horizontal axis represents polarization angle. As shown in the figure, the luminance value varies depending on the polarization angle. When the polarization angle is θ ｐоｌ The luminance value I ｐоｌ is expressed by the following formula: ｐоｌ = A × sin(2θ ｐоｌ ) +B×cos(2θ ｐоｌ ) + C Equation 17 The coefficients A, B, and C in the above equation are expressed by equations 3 to 5 with R removed.

[0096] Furthermore, in plot 501 where the polarization angle is φ, the luminance value is maximum, and the difference between this maximum value and C corresponds to the polarization component ρ. In plot 502, the luminance value is minimum, and the difference between this minimum value and C also corresponds to the polarization component ρ. Therefore, the polarization component ρ can be expressed by Equation 6, with R removed.

[0097] The image sensor 200 uses the wire grid polarizer 330 to capture I ０ , I ４５ , I ９０ , I １３５ Therefore, the polarization component ρ can be calculated from equations 3 to 6.

[0098] Here, an image sensor 200 that performs the processes subsequent to the noise reduction process without calculating the noise estimation amount, the false color amount, and the NR intensity is considered as a comparative example.

[0099] 9 is a graph showing an example of the relationship between luminance values ​​and polarization angles before noise reduction processing in a comparative example. In the figure, a shows an example of the relationship between luminance values ​​of R pixels and B pixels and polarization angles, and b shows an example of the relationship between luminance values ​​of G pixels and polarization angles. The difference between the fine dotted lines and the solid lines in a and b in the figure indicates noise components. This also applies to the following figures.

[0100] Here, it is assumed that there is no difference in sensitivity among the R, G, and B pixels. In this case, as shown in a and b in the figure, the noise components in the pixel signals of the R and B pixels are about the same as the noise component in the pixel signal of the G pixel.

[0101] 10 is a graph showing an example of the relationship between the luminance value and the polarization angle after white balance correction in a comparative example, where "a" in the figure shows an example of the relationship between the luminance value and the polarization angle of the R pixel and the B pixel, and "b" in the figure shows an example of the relationship between the luminance value and the polarization angle of the G pixel.

[0102] Generally, in white balance correction, the luminance value of the G pixel is used as a reference, and the luminance values ​​of the R pixel and the B pixel are corrected according to this reference by the white balance gains WBG_R and WBG_B. On the other hand, the luminance value of the G pixel is not corrected.

[0103] Therefore, as shown in a and b in the figure, after white balance correction, the noise components in the pixel signals of the R and B pixels are enlarged and become larger than the noise components in the pixel signal of the G pixel.

[0104] 11 is a graph showing an example of the relationship between the luminance value and the polarization angle after polarization signal processing in a comparative example, where "a" in the figure shows an example of the relationship between the luminance value and the polarization angle of the R pixel and the B pixel, and "b" in the figure shows an example of the relationship between the luminance value and the polarization angle of the G pixel.

[0105] As shown in a and b in the figure, a difference in the noise component occurs due to the white balance correction in the previous stage, and therefore the polarization component ρ Ｒ , ρ Ｂ and the polarization component ρ Ｇ As a result, in the comparative example, false colors occur in the image after polarization signal processing.

[0106] Furthermore, development processing causes R and B noise components to become larger than G noise components. This is because in a Quad Bayer array or Bayer array, the number of G pixels is twice the number of R or B pixels.

[0107] As described above, in the comparative example, white balance correction and development processing cause a difference between the R and B noise components and the G noise component, and this difference causes false colors. Note that the influence of white balance correction on false colors tends to be greater than the influence of development processing.

[0108] Furthermore, in addition to white balance correction and development processing, if there is a difference in sensitivity among the R, G, and B pixels, false colors also occur due to the difference in sensitivity. However, the influence of white balance correction and development processing on false colors tends to be greater than the influence of sensitivity differences.

[0109] In the comparative example, false colors can be suppressed by increasing the NR intensity, but excessive noise reduction processing can result in a loss of detail and, conversely, a deterioration in image quality, which is not preferable.

[0110] In contrast, according to the first embodiment, which calculates the noise estimate, the amount of false color, and the noise reduction intensity, the R and B noise reduction intensities are increased according to the amount of false color, so that false color can be suppressed with a minimum of appropriate noise reduction processing, thereby improving the image quality of the image after polarization signal processing.

[0111] 12 is a diagram showing an example of setting an ROI and an example of screen division according to the first embodiment of the present technology, where a in the figure shows an example of setting an ROI, and b in the figure shows an example of screen division.

[0112] As shown in the example of a in the figure, it is assumed that ROIs 511, 512, and 513 are set in a reflection-suppressed image 510. ROIs 511 and 512 are areas in which houses and people are captured, and ROI 513 is an area in which the sun is captured.

[0113] The false color estimation processing unit 252 calculates statistics (sums and averages) of luminance values ​​within each of the ROIs 511, 512, and 513. The ROI 513 has a relatively high luminance value because it contains the sun. In such an area, false color is less likely to occur, and even if false color does occur, it is less noticeable. Therefore, the false color estimation processing unit 252 selects the ROIs 511 and 512, whose luminance value statistics are equal to or less than a threshold, as calculation targets, and excludes the ROI 513, whose luminance value statistics exceed the threshold, from the calculation target.

[0114] Furthermore, as shown in the example of b in the figure, the false color estimation processor 252 can divide the reflection-suppressed image 520 into multiple areas and determine whether or not to include each area in the calculation. The dotted lines in b in the figure indicate the boundaries of the areas.

[0115] The false color estimation processor 252 calculates the statistics (total and average) of the luminance values ​​within each area. The upper right area has a relatively high luminance value because it contains the sun. For this reason, the false color estimation processor 252 selects areas whose luminance value statistics are equal to or less than a threshold as areas to be calculated, and excludes areas whose luminance value statistics exceed the threshold (such as the upper right area) from the calculation targets.

[0116] As described above, the false color estimation processor 252 can also perform calculations on the entire image.

[0117] 13 is a graph showing an example of the relationship between the amount of false color and the estimated amount of noise and the NR intensity according to the first embodiment of the present technology. In the figure, "a" is a graph showing an example of the relationship between the amount of false color and the NR intensity. The vertical axis of "a" in the figure indicates the NR intensity, and the horizontal axis indicates the amount of false color estimated by the false color estimation processing unit 252. In the figure, "b" is a graph showing an example of the relationship between the estimated amount of noise and the NR intensity. The vertical axis of "b" in the figure indicates the NR intensity, and the horizontal axis indicates the estimated amount of noise.

[0118] The NR intensity calculation unit 253 uses only the amount of false color or both the amount of false color and the estimated amount of noise when calculating the NR intensity.

[0119] As illustrated in FIG. 10A, the NR intensity calculation unit 253 calculates the false color amount FC ｅｓｔ Based on this, the NR intensity Deg can be calculated by the following linear equation: Deg = a × FC ｅｓｔ +b Equation 18 In the above equation, a and b are predetermined coefficients.

[0120] Furthermore, when both the amount of false color and the estimated amount of noise are used, the NR intensity calculation unit 253 calculates the NR intensity using Equation 1, and sets it as Deg_1.

[0121] Then, as illustrated in FIG. 1B, the NR intensity calculation unit 253 calculates the noise estimation amount N ｅｓｔ Based on this, the NR intensity Deg_2 is calculated by the following linear equation: Deg_2=c×N ｅｓｔ +d Equation 19 In the above equation, c and d are predetermined coefficients.

[0122] Then, the NR strength calculation unit 253 calculates the statistical quantity (average, etc.) of the NR strengths Deg_1 and Deg_2 as the final NR strength.

[0123] 14 is a flowchart showing an example of the operation of the image processing system according to the first embodiment of the present technology. This operation is started, for example, when a predetermined application for image capture is executed.

[0124] The image capturing device 110 generates an input image (step S901) and performs noise estimation processing to calculate a noise estimation amount (step S902). The image capturing device 110 then performs false color estimation processing to estimate the amount of false color (step S903) and calculates NR intensity (step S904). The image capturing device 110 then performs noise reduction processing (step S905), shading correction, digital gain processing, and white balance correction (step S906). The image capturing device 110 then performs development processing (step S907), and the information processing device 120 performs polarization signal processing (step S908). After step S908, the image processing system 100 ends its image processing operations.

[0125] When a plurality of images are captured consecutively, steps S901 to S908 are repeatedly executed in synchronization with a vertical synchronization signal.

[0126] As described above, according to the first embodiment of the present technology, the image processing system 100 calculates the amount of false color and calculates the NR intensity based on the amount of false color, thereby suppressing false color through noise reduction processing with an appropriate NR intensity, thereby improving the image quality after polarization signal processing.

[0127] [Modification] In the first embodiment described above, the image sensor 200 calculates the amount of noise from the level of the OPB pixels, but in this configuration, it is necessary to arrange the OPB pixels within the image sensor 200. The image sensor 200 in this modification of the first embodiment differs from the first embodiment in that it calculates the amount of noise from temperature and at least one of analog gain, digital gain, and exposure time.

[0128] 15 is a block diagram showing an example of a configuration of an image sensor 200 according to a modification of the first embodiment of the present technology. The image sensor 200 according to the modification of the first embodiment differs from the first embodiment in that it further includes a temperature measurement unit 280.

[0129] The temperature measurement unit 280 measures the temperature inside the image sensor 200 and transmits it as a measured temperature T to the image processing unit 250 .

[0130] Although the temperature measurement unit 280 is disposed inside the image sensor 200, it may be disposed outside the image sensor 200 as long as it is within the imaging device 110.

[0131] FIG. 16 is a diagram for explaining a method for estimating the amount of noise in a modified example of the first embodiment of the present technology.

[0132] In the first modification of the first embodiment, the noise estimation processing unit 251 holds a table as shown in the figure, in which sensor setting values ​​and noise amounts are stored in association with each other for each of a plurality of temperature ranges.

[0133] Here, the sensor setting value is the product (addition in decibel notation) of the analog gain, digital gain, and the ratio of the exposure time to the reference time. The larger the sensor setting value, the greater the amount of noise.

[0134] Although the noise estimation processing unit 251 uses all of the analog gain, digital gain, and exposure time, it is not limited to this configuration and may use only some of these.

[0135] As shown in the figure, for each temperature range such as "less than T1" or "greater than T and less than T2", sensor setting values ​​such as "G1" and "G2" and noise amounts such as "N1" and "N2" are listed in one-to-one correspondence.

[0136] The noise estimation processing unit 251 refers to the table to obtain the amount of noise corresponding to the measured temperature and the sensor setting value. Then, the noise estimation processing unit 251 calculates the amount of noise as N ｃａｌｃ＿Ｒ and N ｃａｌｃ＿Ｂare substituted into Equation 1 and Equation 2 together with the digital gain and the white balance gain to calculate the noise estimation amount.

[0137] As described above, the noise estimation processing unit 251 calculates the noise estimation amount based on the temperature, the sensor setting values ​​(analog gain, digital gain, and exposure time), and the white balance gain, thereby eliminating the need for OPB pixels.

[0138] As described above, according to the modification of the first embodiment of the present technology, the noise estimation processing unit 251 calculates the noise estimation amount based on the temperature, the sensor setting values ​​(analog gain, digital gain, and exposure time), and the white balance gain, thereby making it possible to reduce OPB pixels.

[0139] 2. Second Embodiment In the first embodiment described above, the amount of false color and the noise reduction intensity are calculated before white balance correction, but these calculations can also be performed after white balance correction. The image processing system 100 in this second embodiment differs from the first embodiment in that the amount of false color and the noise reduction intensity are calculated after white balance correction.

[0140] FIG. 17 is a block diagram showing an example configuration of the image processing unit 250 and the information processing device 120 according to the second embodiment of the present technology.

[0141] The image processing unit 250 of the second embodiment differs from that of the first embodiment in that it includes a noise detection processing unit 261 and a development processing unit 262 instead of the noise estimation processing unit 251, the noise reduction processing unit 254, and the development processing unit 258.

[0142] In the second embodiment, the input image is input to a shading correction unit 255. Then, a white balance correction unit 257 outputs a WB-corrected image to a noise detection processing unit 261, a false color estimation processing unit 252, and a development processing unit 262.

[0143] The noise detection processing unit 261 detects the amount of noise after WB correction. For example, the noise detection processing unit 261 calculates the standard deviation of the polarization angle φ corresponding to the polarization component in each pixel block of R, G, and B in an area where the luminance is flat. Then, the noise detection processing unit 261 detects the difference between the standard deviation of R and B and the standard deviation of G as the amount of noise of R and B. The noise detection processing unit 261 calculates the detected amount of noise as N ｄｅｔ to the false color estimation processing unit 252. In this way, since the OPB pixels are not used in the second embodiment, the OPB pixels in the pixel array unit 220 can be reduced.

[0144] When the polarizing plate 270 is used instead of the wire grid polarizer 330, the polarization components ρ Ｂ , ρ Ｇ and ρ Ｂ is calculated.

[0145] Furthermore, the noise detection processing unit 261 can also use other criteria when detecting the amount of noise from the polarization components. For example, if the image processing unit 250 has also performed AE (auto exposure) in a previous stage, the noise detection processing unit 261 determines the brightness of the image by referring to the gain used in AE. Generally, the darker the image, the greater the amount of noise, so the noise detection processing unit 261 corrects the amount of noise to a larger value as the image becomes darker.

[0146] Then, the false color estimation processing unit 252 calculates the noise estimation amount N ｅｓｔ Instead of the noise amount N ｄｅｔ The NR intensity calculation unit 253 calculates the amount of false color by the same processing as in the first embodiment using the NR intensity calculation unit 253, and outputs the calculated amount of false color to the development processing unit 262. The NR intensity calculation unit 253 calculates the NR intensity by the same processing as in the first embodiment, and outputs the calculated amount to the development processing unit 262.

[0147] The development processing unit 262 performs development processing on the WB corrected image as well as noise reduction processing of NR intensity. For example, the development processing unit 262 applies a low-pass filter to the WB corrected image and then generates a correlated wideband signal. The processing of applying this low-pass filter corresponds to noise reduction processing.

[0148] Although the development processing unit performs noise reduction processing on the WB corrected image (i.e., the mosaic image), it is also possible to perform noise reduction processing on the mosaic image after development processing instead.

[0149] FIG. 18 is a flowchart showing an example of the operation of the image processing system according to the second embodiment of the present technology.

[0150] The image capturing device 110 generates an input image (step S901) and performs shading correction, digital gain processing, and white balance correction (step S906). The image capturing device 110 then performs noise detection (step S911) and false color estimation (step S903). The image capturing device 110 then calculates NR intensity (step S904) and performs development processing and noise reduction processing (step S912). The information processing device 120 then performs polarization signal processing (step S908).

[0151] As described above, according to the second embodiment of the present technology, the image processing system 100 detects the amount of noise and calculates the amount of false color and NR intensity after white balance correction, thereby eliminating the need to calculate the estimated amount of noise.

[0152] 3. Third Embodiment In the second embodiment described above, the image processing system 100 detects the amount of noise and calculates the amount of false color and the noise reduction intensity before the development process, but these processes can also be performed after the development process. The image processing system 100 in this second embodiment differs from the second embodiment in that the detection of the amount of noise and the calculation of the amount of false color and the noise reduction intensity are performed after the development process.

[0153] 19 is a block diagram showing an example configuration of an image processing unit 250 and an information processing device 120 according to the third embodiment of the present technology. In the third embodiment, the image processing unit 250 does not include a noise detection processing unit 261, a false color estimation processing unit 252, and an NR intensity calculation unit 253, and instead includes a development processing unit 258 instead of the development processing unit 262. The information processing device 120 further includes a noise detection processing unit 122, a false color estimation processing unit 123, and an NR intensity calculation unit 124, and instead of the polarization signal processing unit 121, a polarization signal processing unit 125.

[0154] In the third embodiment, the white balance correction unit 257 outputs the WB corrected image to the development processing unit 258. The development processing unit 258 performs development processing on the WB corrected image and outputs it as an output image to the noise detection processing unit 122, the false color estimation processing unit 123, and the polarization signal processing unit 125.

[0155] The noise detection processing unit 122 detects the amount of noise after development processing by the same processing as in the second embodiment, and outputs the amount of noise to the false color estimation processing unit 123 .

[0156] The false color estimation processing unit 123 calculates the amount of false color by the same processing as in the second embodiment, and outputs the amount of false color to the NR intensity calculation unit 124 .

[0157] The NR intensity calculation unit 124 calculates the NR intensity by the same processing as in the first embodiment, and outputs it to the polarization signal processing unit 125 .

[0158] The polarization signal processing unit 125 performs polarization signal processing as well as noise reduction processing of NR intensity on the output image.

[0159] As illustrated in the same figure, the information processing device 120 detects the amount of noise and calculates the amount of false color and NR intensity after the development process, so the noise detection processing unit 261, false color estimation processing unit 252, and NR intensity calculation unit 253 within the imaging device 110 can be reduced.

[0160] FIG. 20 is a flowchart showing an example of the operation of the image processing system 100 according to the third embodiment of the present technology.

[0161] The image capturing apparatus 110 generates an input image (step S901), performs shading correction, digital gain processing, and white balance correction (step S906), and then performs development processing (step S907).

[0162] The information processing device 120 performs noise detection processing (step S911), performs false color estimation processing (step S903), calculates NR intensity (step S904), and performs polarization signal processing and noise reduction processing (step S931).

[0163] As described above, according to the third embodiment of the present technology, the information processing device 120 detects the amount of noise and calculates the amount of false color and the NR intensity after the development process, which makes it possible to reduce the noise detection processing unit 261, the false color estimation processing unit 252, and the NR intensity calculation unit 253 in the imaging device 110.

[0164] 4. Fourth Embodiment In the first embodiment described above, the image processing unit 250 uses the calculated NR intensity in a subsequent stage, but it can also feed it back to a previous stage. The image processing system 100 in this fourth embodiment differs from the first embodiment in that the calculated NR intensity is fed back to a previous stage.

[0165] FIG. 21 is a block diagram showing an example configuration of the image processing unit 250 and the information processing device 120 according to the fourth embodiment of the present technology.

[0166] The image processing unit 250 in the fourth embodiment differs from that in the first embodiment in that the noise estimation processing unit 251, the false color estimation processing unit 252, and the NR intensity calculation unit 253 are not provided.

[0167] The information processing device 120 according to the fourth embodiment also differs from that according to the first embodiment in that it further includes a false color detection processing unit 126 and an NR intensity calculation unit 124 .

[0168] In the fourth embodiment, the noise reduction processing unit 254 receives an input image and the NR intensity from the NR intensity calculation unit 124. If the input image to be processed by the noise reduction processing unit 254 is the current frame and the input image generated immediately before that is the previous frame, the NR intensity calculated from the output image corresponding to the previous frame is applied to the noise reduction processing for the current frame.

[0169] The processing contents of the noise reduction processing unit 254, shading correction unit 255, digital gain processing unit 256, white balance correction unit 257, development processing unit 258, and polarization signal processing unit 121 are the same as those in the first embodiment, except that the polarization signal processing unit 121 also outputs the processed image to the false color detection processing unit 126.

[0170] The false color detection processing unit 126 calculates the amount of false color generated in the processed image. As described above, polarization signal processing generates processed images such as an average image, a polarization component image, a polarization degree image, a reflection-enhanced image, a reflection-suppressed image, and a polarization direction image. Of these, various images other than the average image contain polarization components.

[0171] For example, in polarization signal processing, an average image and an image containing polarization components (a polarization component image, a polarization degree image, a reflection-enhanced image, a reflection-suppressed image, or a polarization direction image) are generated and input to the false color detection processing unit 126. The false color detection processing unit 126 calculates (in other words, detects) the difference in hue between the average image and the image containing polarization components as the amount of false color.

[0172] Alternatively, if a specific subject S in the average image does not have a specific color (in other words, is white), but the subject S in the image containing the polarized light component has a color, the false color detection processing unit 126 detects that color as a false color.

[0173] The false color detection processing unit 126 calculates the detected amount of false color as FC ｄｅｔ The NR strength calculation unit 124 outputs the estimated FC ｅｓｔ FC detected instead of ｄｅｔThe NR intensity is calculated by the same process as in the first embodiment using the above and is fed back to the noise reduction processing unit 254 .

[0174] The false color detection processing unit 126 is an example of a false color calculation unit described in the claims.

[0175] As shown in the figure, the NR intensity calculation unit 124 feeds back the NR intensity, and the false color detection processing unit 126 detects the amount of false color generated in the processed image, so there is no need to calculate the noise estimation amount.

[0176] FIG. 22 is a flowchart showing an example of the operation of the image processing system 100 according to the fourth embodiment of the present technology.

[0177] The image capturing device 110 generates an input image in synchronization with a vertical synchronization signal (step S901), performs noise reduction processing (step S905), performs shading correction, digital gain processing, and white balance correction (step S906), and performs development processing (step S907).

[0178] The information processing device 120 performs polarization signal processing (step S908). Then, the information processing device 120 performs false color detection processing to detect the amount of false color (step S941), calculates the NR intensity (step S904), and then determines whether to end image capture (step S942).

[0179] If imaging is to be continued (step S942: No), the image processing system 100 repeats step S901 and subsequent steps. On the other hand, if imaging is to be ended (step S942: Yes), the image processing system 100 ends the operation for image processing.

[0180] As described above, according to the fourth embodiment of the present technology, the false color detection processing unit 126 detects the amount of false color that has occurred in the processed image, and therefore there is no need to calculate the noise estimation amount.

[0181] [First Modification] In the fourth embodiment described above, the image processing unit 250 performs noise reduction processing before development processing, but noise reduction processing can also be performed during development processing. The image processing unit 250 in this first modification of the fourth embodiment differs from the fourth embodiment in that it performs noise reduction processing during development processing.

[0182] 23 is a block diagram showing an example configuration of an image processing unit 250 and an information processing device 120 according to a first modified example of the fourth embodiment of the present technology. The image processing unit 250 according to the first modified example of the fourth embodiment differs from the fourth embodiment in that it includes a development processing unit 262 instead of the noise reduction processing unit 254 and the development processing unit 258.

[0183] In a first modification of the fourth embodiment, an input image is input to the shading correction unit 255. The development processing unit 262 performs development processing on the WB corrected image and noise reduction processing of the NR intensity. The NR intensity calculation unit 124 feeds back the NR intensity to the development processing unit 262.

[0184] As described above, according to the first modified example of the fourth embodiment of the present technology, the development processing unit 262 performs the noise reduction processing together with the development processing, and therefore the noise reduction processing unit 254 is not necessary.

[0185] [Second Modification] In the fourth embodiment described above, the image processing unit 250 performs noise reduction processing before development processing, but noise reduction processing can also be performed during polarization signal processing. The image processing unit 250 in this second modification of the fourth embodiment differs from the fourth embodiment in that it performs noise reduction processing during polarization signal processing.

[0186] 24 is a block diagram showing an example configuration of an image processing unit 250 and an information processing device 120 according to a second modified example of the fourth embodiment of the present technology. The image processing unit 250 according to the second modified example of the fourth embodiment differs from the fourth embodiment in that it does not include a noise reduction processing unit 254. Furthermore, the information processing device 120 according to the second modified example of the fourth embodiment differs from the first embodiment in that it includes a polarization signal processing unit 125 instead of the polarization signal processing unit 121.

[0187] In a second modification of the fourth embodiment, an input image is input to a shading correction unit 255. The polarization signal processing unit 125 performs polarization signal processing on the output image as well as noise reduction processing of the NR intensity. The NR intensity calculation unit 124 feeds back the NR intensity to the polarization signal processing unit 125.

[0188] In Fig. 21, the image processing unit 250 performs noise reduction processing before shading correction, and in Fig. 23, noise reduction processing is performed during development processing. In Fig. 24, noise reduction processing is performed by the information processing device 120 during polarization signal processing. All or only some of these three noise reduction processes can be performed.

[0189] As described above, according to the second modification of the fourth embodiment of the present technology, the polarization signal processing unit 125 performs the noise reduction processing together with the polarization signal processing, and therefore the noise reduction processing unit 254 is not necessary.

[0190] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology having the same name correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist thereof.

[0191] The processing procedures described in the above embodiments may be considered as a method having a series of these procedures, or as a program for causing a computer to execute the series of procedures, or as a recording medium for storing the program. Examples of such a recording medium include a CD (Compact Disc), an MD (MiniDisc), a DVD (Digital Versatile Disc), a memory card, and a Blu-ray (registered trademark) Disc.

[0192] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0193] The present technology may also be configured as follows: (1) An image processing system comprising: a false color amount calculation unit that calculates an amount of false color after performing polarization signal processing related to polarization components; and a noise reduction intensity calculation unit that calculates a degree of noise reduction processing for a color image captured using a polarization filter as a noise reduction intensity based on the amount of false color. (2) The image processing system according to (1), further comprising: a noise estimation processing unit that calculates an amount of noise after white balance correction as a noise estimate and outputs the amount of noise to the false color amount calculation unit; a noise reduction processing unit that performs the noise reduction processing of the noise reduction intensity on the color image and outputs it as a noise reduced image; a white balance correction unit that performs the white balance correction on the noise reduced image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the amount of false color from the noise estimate. (3) The image processing system according to (2), wherein the color image includes levels of OPB pixels, and the noise estimation processor calculates the noise estimation amount based on statistics of the levels of the OPB pixels and a white balance gain used in the white balance correction. (4) The image processing system according to (2), wherein the noise estimation processor calculates the noise estimation amount based on a temperature measured in the image processing system, at least one of an analog gain, a digital gain, and an exposure time used in the image processing system, and a white balance gain used in the white balance correction.(5) The image processing system according to (1), further comprising: a white balance correction unit that performs white balance correction on the color image and outputs the result as a white balance corrected image; a noise detection processing unit that detects the amount of noise after the white balance correction and outputs the result to a false color amount calculation processing unit; a development processing unit that performs the noise reduction processing of the noise reduction intensity on the white balance corrected image together with a development processing and outputs the result as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the false color amount from the detected amount of noise. (6) The image processing system according to (1), further comprising: a white balance correction unit that performs white balance correction on the color image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; a noise detection processing unit that detects the amount of noise after development processing and outputs the result to a false color amount calculation processing unit; and a polarization signal processing unit that performs the polarization signal processing and the noise reduction processing of the noise reduction intensity on the output image, wherein the false color amount calculation unit calculates the amount of false color from the detected amount of noise. (7) The image processing system according to (1), further comprising: a white balance correction unit that performs white balance correction on the color image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image and outputs it as a processed image, wherein the false color amount calculation unit calculates the amount of false color generated in the processed image. (8) The image processing system according to (7), further comprising a noise reduction processing unit that performs the noise reduction processing of the noise reduction intensity on the color image and outputs the result to the white balance correction unit. (9) The image processing system according to (7) or (8), wherein the development processing unit further performs the noise reduction processing of the noise reduction intensity together with the development processing.(10) The image processing system according to any one of (7) to (9), wherein the polarization signal processing unit further performs the noise reduction processing of the noise reduction intensity together with the polarization signal processing. (11) An image processing device comprising: a false color amount calculation unit that calculates the amount of false color after the polarization signal processing, which is processing related to polarization components, and a noise reduction intensity calculation unit that calculates the degree of noise reduction processing for a color image captured using a polarization filter based on the false color amount. (12) The image processing device according to (11), further comprising: a noise estimation processing unit that calculates a noise amount after white balance correction as a noise estimate and outputs the calculated amount to the false color amount calculation unit; a noise reduction processing unit that performs the noise reduction processing of the noise reduction intensity on the color image and outputs it as a noise reduced image; a white balance correction unit that performs the white balance correction on the noise reduced image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the false color amount from the noise estimate. (13) The image processing device according to (12), (14) The image processing device according to (12), wherein the noise estimation processing unit calculates the noise estimation amount based on a temperature measured within the image processing device, at least one of an analog gain, a digital gain, and an exposure time used in the image processing device, and a white balance gain used in the white balance correction.(15) The image processing device according to (11), further comprising: a white balance correction unit that performs the white balance correction on the color image and outputs it as a white balance corrected image; the noise detection processing unit that detects an amount of noise after the white balance correction and outputs the detected amount of noise to a false color amount calculation processing unit; a development processing unit that performs the noise reduction processing of the white balance corrected image using the noise reduction intensity together with a development processing, and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the amount of false color from the detected amount of noise. (16) An image processing method, comprising: a false color amount calculation step that calculates the amount of false color after polarization signal processing that is processing related to polarization components, and a noise reduction intensity calculation step that calculates a degree of noise reduction processing for a color image captured using a polarization filter as the noise reduction intensity based on the false color amount. (17) A program for causing a computer to execute: a false color amount calculation procedure for calculating the amount of false color after polarization signal processing, which is processing related to polarization components; and a noise reduction strength calculation procedure for calculating the degree of noise reduction processing for a color image captured using a polarizing filter as noise reduction strength based on the false color amount.

[0194] 100 Image processing system 110 Imaging device 111 Optical unit 112 DSP circuit 113 Operation unit 114 Bus 115 Frame memory 116 Storage unit 117 Power supply unit 118 Communication unit 120 Information processing device 121, 125 Polarization signal processing unit 122, 261 Noise detection processing unit 123, 252 False color estimation processing unit 124, 253 NR intensity calculation unit 126 False color detection processing unit 200 Image sensor 211 Vertical drive circuit 212 Timing control unit 213 DAC 214 Column signal processing unit 215 Horizontal transfer control unit 220 Pixel array unit 221 VOPB area 222 HOPB area 223 Light receiving area 230 OPB 241, 242, 243 Pixel block 250 Image processing unit 251: Noise estimation processing unit 254: Noise reduction processing unit 255: Shading correction unit 256: Digital gain processing unit 257: White balance correction unit 258, 262: Development processing unit 270: Polarizing plate 280: Temperature measurement unit 311 to 314: On-chip lens 320: Color filter 330: Wire grid polarizer 331 to 334: Polarizing filter 341 to 344: Photodiode

Claims

1. A pseudo-color amount calculation unit that calculates the amount of pseudo-color after performing polarization signal processing, which is processing related to polarization components, and a noise reduction intensity calculation unit that calculates the degree of noise reduction processing for a color image captured using a polarization filter as a noise reduction intensity based on the amount of pseudo-color. An image processing system comprising:

2. A noise estimation processing unit that calculates the amount of noise after white balance correction as a noise estimation amount and outputs it to the pseudo-color amount calculation unit, a noise reduction processing unit that performs the noise reduction processing of the noise reduction intensity on the color image and outputs it as a noise reduction image, a white balance correction unit that performs the white balance correction on the noise reduction image and outputs it as a white balance correction image, a development processing unit that performs development processing on the white balance correction image and outputs it as an output image, and a polarization signal processing unit that performs the polarization signal processing on the output image. Further comprising: The pseudo-color amount calculation unit calculates the amount of pseudo-color from the noise estimation amount. The image processing system according to claim 1.

3. The color image includes the level of an OPB (Optical Black) pixel, and the noise estimation processing unit calculates the noise estimation amount based on the statistical amount of the level of the OPB pixel and the white balance gain used in the white balance correction. The image processing system according to claim 2.

4. The noise estimation processing unit calculates the noise estimation amount based on the temperature measured within the image processing system and at least one of the analog gain, digital gain, and exposure time used in the image processing system and the white balance gain used in the white balance correction. The image processing system according to claim 2.

5. A white balance correction unit that performs white balance correction on the color image and outputs it as a white balance corrected image; a noise detection processing unit that detects the amount of noise after the white balance correction and outputs it to a false color amount calculation processing unit; a development processing unit that performs the noise reduction processing with the noise reduction intensity together with the development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image. The false color amount calculation unit calculates the false color amount from the detected amount of noise. The image processing system according to claim 1.

6. A white balance correction unit that performs white balance correction on the color image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; a noise detection processing unit that detects the amount of noise after the development processing and outputs it to a false color amount calculation processing unit; and a polarization signal processing unit that performs the noise reduction processing with the noise reduction intensity together with the polarization signal processing on the output image. The false color amount calculation unit calculates the false color amount from the detected amount of noise. The image processing system according to claim 1.

7. A white balance correction unit that performs white balance correction on the color image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image and outputs it as a processed image. The false color amount calculation unit calculates the false color amount generated in the processed image. The image processing system according to claim 1.

8. The image processing system according to claim 7, further comprising a noise reduction processing unit that performs the noise reduction processing with the noise reduction intensity on the color image and outputs it to the white balance correction unit.

9. The image processing system according to claim 7, wherein the development processing unit further performs the noise reduction processing with the noise reduction intensity together with the development processing.

10. The image processing system according to claim 7, wherein the polarization signal processing unit further performs the noise reduction processing of the noise reduction intensity together with the polarization signal processing.

11. An image processing apparatus comprising: a false color amount calculation unit that calculates a false color amount after performing polarization signal processing, which is processing related to polarization components; and a noise reduction intensity calculation unit that calculates the degree of noise reduction processing for a color image captured using a polarization filter as a noise reduction intensity based on the false color amount.

12. The image processing apparatus according to claim 11, further comprising: a noise estimation processing unit that calculates the amount of noise after white balance correction as a noise estimation amount and outputs it to the false color amount calculation unit; a noise reduction processing unit that performs the noise reduction processing of the noise reduction intensity on the color image and outputs it as a noise reduction image; a white balance correction unit that performs the white balance correction on the noise reduction image and outputs it as a white balance corrected image; a development processing unit that performs development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image, wherein the false color amount calculation unit calculates the false color amount from the noise estimation amount.

13. The image processing apparatus according to claim 12, wherein the color image includes the level of an OPB pixel, and the noise estimation processing unit calculates the noise estimation amount based on the statistical amount of the level of the OPB pixel and the white balance gain used in the white balance correction.

14. The image processing apparatus according to claim 12, wherein the noise estimation processing unit calculates the noise estimation amount based on at least one of the temperature measured in the image processing apparatus, the analog gain, the digital gain, and the exposure time used in the image processing apparatus, and the white balance gain used in the white balance correction.

15. A white balance correction unit that performs the white balance correction on the color image and outputs it as a white balance corrected image; a noise detection processing unit that detects the amount of noise after the white balance correction and outputs it to a false color amount calculation processing unit; a development processing unit that performs the noise reduction processing with the noise reduction intensity together with development processing on the white balance corrected image and outputs it as an output image; and a polarization signal processing unit that performs the polarization signal processing on the output image. The false color amount calculation unit calculates the false color amount from the detected amount of noise. The image processing apparatus according to claim 11.

16. A false color amount calculation procedure for calculating a false color amount after performing polarization signal processing, which is processing related to a polarization component; and a noise reduction intensity calculation procedure for calculating the degree of noise reduction processing for a color image captured using a polarization filter as a noise reduction intensity based on the false color amount. An image processing method comprising the steps of:

17. A program for causing a computer to execute a false color amount calculation procedure for calculating a false color amount after performing polarization signal processing, which is processing related to a polarization component; and a noise reduction intensity calculation procedure for calculating the degree of noise reduction processing for a color image captured using a polarization filter as a noise reduction intensity based on the false color amount.

Images