Image processing device, image processing method, and imaging device

Abstract

An invalid pixel detection unit 32 detects invalid pixels from a non-polarization image and a plurality of polarization images having different polarization directions, the non-polarization image and the polarization images being obtained by imaging using a polarization imaging unit 20. For example, the invalid pixel detection unit 32 detects, from the non-polarization image, a saturated pixel having a pixel value larger than a preset saturation detection threshold value, and a black crushed pixel having a pixel value smaller than a preset black crushing detection threshold value as the invalid pixels. A polarization information generation unit 33 performs polarization information generation processing on the basis of the non-polarization image and the polarization images, switches the polarization information generation processing according to the result of detection of the invalid pixels in the invalid pixel detection unit 32, and generates polarization information without using the invalid pixels, thereby being able to acquire correct polarization information.

Description

[Technical field]

[0001] This technique relates to an image processing device, an image processing method, and an imaging device, and makes it possible to acquire polarization information with high accuracy. [Background technology]

[0002] Conventionally, a method for acquiring a polarized image using a polarization imaging section and a polarizing filter has been disclosed. For example, Patent Document 1 discloses a method for arranging a patterned polarizer and a color mosaic filter in front of an imaging element to capture an image of a subject, and acquiring color information and polarization information based on the acquired polarized image. Patent Document 2 discloses that by using not only a polarized image showing a polarization component but also a non-polarized image without a reduction in the amount of light, it is possible to acquire polarization information with higher accuracy than when the polarization information is generated based on a polarized image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-290895 A [Patent Document 2] International Publication No. 2018 / 074064 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, in order to obtain accurate polarization information by using non-polarization pixels, it is necessary that both the two non-polarization pixels and the two polarizing pixels hold valid pixel values. If the non-polarization pixels become saturated or the polarizing pixels become blacked out, there is a risk that the obtained polarization information will contain errors.

[0005] Therefore, this technique provides an image processing device, an image processing method, and an imaging device that can acquire correct polarization information. [Means for solving the problem]

[0006] The first aspect of this technology is: an invalid pixel detection unit that detects saturated pixels and crushed black pixels as invalid pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using the polarization image capturing unit; a polarization information generating unit that performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating the polarization information depending on a detection result of the invalid pixel by the invalid pixel detecting unit; The present invention relates to an image processing device comprising:

[0007] In this technology, the invalid pixel detection unit detects invalid pixels from a non-polarized image obtained by performing imaging using the polarization imaging unit and from a plurality of polarized images having different polarization directions. For example, the invalid pixel detection unit detects, from the non-polarized image, saturated pixels having pixel values ​​larger than a preset saturation detection threshold and black crushed pixels having pixel values ​​smaller than a preset black crushed detection threshold as invalid pixels. The saturation detection threshold and the black crushed detection threshold may be set according to the characteristics of an image obtained by performing imaging using the polarization imaging unit. For example, the black crushed detection threshold is set to be larger than the minimum output value of the polarization imaging unit by a first level, and the saturation detection threshold is set to be smaller than the maximum output value of the polarization imaging unit by a second level smaller than the first level.

[0008] The polarization information generating unit performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating polarization information depending on the result of detection of an invalid pixel by the invalid pixel detecting unit. For example, when a pixel of the unpolarized image is detected as an invalid pixel, the polarization information generating unit generates polarization information using the pixel of the unpolarized image and at least pixels of two or more polarized images that are not detected as invalid pixels and whose polarization directions are not perpendicular to the pixel of the unpolarized image. The multiple polarized images are three or more polarized images with different polarization directions, and the polarization information generating unit generates polarization information using the pixel of the unpolarized image and the pixel of at least two or more polarized images that are not detected as invalid pixels. Furthermore, when an invalid pixel is not detected, the polarization information generating unit generates polarization information using the pixel of the unpolarized image and the pixel of the polarized image.

[0009] The second aspect of this technology is detecting, as invalid pixels, saturated pixels and crushed black pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using the polarization image capturing unit, using an invalid pixel detection unit; A polarization information generating unit performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating the polarization information depending on a detection result of the invalid pixels by the invalid pixel detecting unit. The present invention relates to an image processing method including the steps of:

[0010] The third aspect of this technology is A program for causing a computer to generate polarization information, a step of detecting saturated pixels and crushed black pixels as invalid pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using a polarization image capturing unit; generating polarization information based on the unpolarized image and the polarized image; a step of switching a generation process of the polarization information in response to a detection result of the invalid pixel; The program is executed by the computer. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of a configuration of a system using an image processing device. [Diagram 2] 3 is a diagram illustrating an example of the configuration of a polarization imaging section. FIG. [Diagram 3] FIG. 2 is a diagram illustrating a configuration of an imaging element. [Figure 4] 1 is a diagram illustrating a pixel configuration of a polarizing filter. [Diagram 5] FIG. 2 is a diagram illustrating an example of the configuration of a demosaic processing unit. [Figure 6] 11A and 11B are diagrams for explaining demosaic processing of polarization pixels. [Figure 7] 10 is a diagram illustrating an example of the relationship (part 1) between the luminance of a non-polarization pixel, the luminance of a polarization pixel, and the angle of the polarization direction. [Figure 8]13 is a diagram illustrating an example of the relationship (part 2) between the luminance of a non-polarization pixel, the luminance of a polarization pixel, and the angle of the polarization direction. FIG. [Figure 9] 4 is a flowchart showing the operation of an image processing unit. [Figure 10] 1 is a diagram illustrating an example of the configuration of a polarization imaging section that generates a color captured image. [Figure 11] This illustrates the relationship between a color mosaic filter and a polarizing filter (two polarization directions). [Figure 12] This illustrates the relationship between a color mosaic filter and a polarizing filter (three polarization directions). DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Configuration and Operation of the Embodiment 2. Other embodiments 3. Application Examples

[0013] 1. Configuration and Operation of the Embodiment 1 shows an example of the configuration of a system using an image processing device. The system 10 includes a polarization image capturing section 20 and an image processing section 30.

[0014] The polarization image capturing section 20 has non-polarization pixels and polarization pixels provided for at least two polarization directions, and generates an image signal of a captured image and outputs it to the image processing section 30.

[0015] 2 illustrates an example of the configuration of the polarization imaging section 20. The polarization imaging section 20 includes a polarization filter 22 disposed on the incident surface of an imaging element 21 such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD).

[0016] 3 illustrates an example of the configuration of an image sensor. The image sensor 21 has a pixel array section 211 in which a plurality of pixels are arranged in an array, for example, a two-dimensional matrix, and a vertical scanning circuit 212 and a horizontal scanning circuit 213 that perform drive control of the pixel array section 211. For ease of explanation, only some of the pixels in the row and column directions are shown in the pixel array section 211.

[0017] Although not shown, each pixel in the pixel array section 211 has a photodiode and a transistor for charge transfer and resetting. Each pixel is connected to a vertical scanning circuit 212 via a reset line and a selection line, and is connected to a horizontal scanning circuit 213 via a signal line.

[0018] The vertical scanning circuit 212 outputs a reset signal to the transistors for resetting the pixels via the reset line to discharge the accumulated charge. After that, the vertical scanning circuit 212 outputs a read signal to the transistors for charge transfer of the polarized pixels and the non-polarized pixels via the selection line to output the charge accumulated during the exposure period from when the reset signal is output to when the read signal is output as a signal current to the signal line. The horizontal scanning circuit 213 performs a process of converting the signal current read from each pixel into a digital pixel signal, a process of adjusting the gain of the pixel signal, and the like, and outputs the processed pixel signal to the image processing unit 30 in the order of pixels in the horizontal direction. The vertical scanning circuit 212 and the horizontal scanning circuit 213 also perform the above-mentioned process for each line.

[0019] 4 shows an example of the pixel configuration of the polarizing filter. The polarizing filter 22 is composed of non-polarizing pixels and polarizing pixels with at least two polarization directions, and the polarizing pixels use photonic liquid crystals or wire grids, etc.

[0020] 4(a) illustrates an example in which polarizing filter 22 is composed of a non-polarizing pixel and three polarizing pixels with different polarization directions. Polarizing filter 22 has a polarizing pixel block, for example, a 2×2 pixel region, and the polarizing pixel block is composed of one non-polarizing pixel and three polarizing pixels with polarization directions of "0°, 90°, 135°", and the polarizing pixel blocks are arranged repeatedly in the horizontal and vertical directions.

[0021] 4(b) illustrates an example in which polarizing filter 22 is composed of a non-polarizing pixel and two polarizing pixels with different polarization directions. Polarizing filter 22 has a polarizing pixel block, for example, a 2×2 pixel region, and the polarizing pixel block is composed of two non-polarizing pixels and two polarizing pixels with polarization directions of "90°, 135°", and the polarizing pixel blocks are repeatedly arranged in the horizontal and vertical directions.

[0022] The polarization imaging section 20 configured in this manner sequentially reads out pixel signals to generate image signals of a captured image composed of polarizing pixels and non-polarizing pixels, and outputs the image signals to the image processing section 30. The polarization imaging section 20 may also control the reset timing of the non-polarizing pixels to set the exposure period of the non-polarizing pixels so that the sensitivity of the non-polarizing pixels is equal to that of the polarizing pixels.

[0023] The image processing unit 30 has a demosaic processing unit 31, an invalid pixel detection unit 32, and a polarization information generation unit 33. The demosaic processing unit 31 may be provided in the polarization imaging unit 20.

[0024] The demosaic processing unit 31 generates a non-polarized image and a polarized image for each polarization direction from the captured image acquired by the polarization imaging unit 20.

[0025] The demosaic processing unit 31 calculates the pixel value of the polarized pixel at the pixel position of the pixel of interest for each polarization direction using polarized pixels at pixel positions of the same polarization direction in the pixel of interest and the pixels surrounding the pixel of interest in the captured image acquired by the polarization imaging unit 20, and non-polarized pixels at pixel positions equivalent to the polarized pixels and the pixel position of the pixel of interest.

[0026] Furthermore, when the pixel interval of non-polarized pixels, which have higher sensitivity than polarized pixels, is equal to or less than the pixel interval of pixels in the same polarization direction, the non-polarized image generated from the non-polarized pixels has a higher resolution than the polarized image generated from pixels in the same polarization direction. Therefore, the demosaic processor 31 assumes that there is a positive correlation between the non-polarized pixels and the polarized pixels, and generates a high-resolution polarized image by compensating for the high-frequency components lost in the polarized image from the non-polarized image. For example, the demosaic processor calculates the pixel value of the polarized pixel at the pixel position of the pixel of interest relative to the pixel average value of the polarized pixels at the pixel position of the pixel of interest using the relationship between the pixel average value of the non-polarized pixels at the pixel positions of pixels in the same polarization direction and the pixel value of the non-polarized pixels at the pixel position of the pixel of interest.

[0027] 5 illustrates an example of the configuration of the demosaic processing unit. The demosaic processing unit 31 has a non-polarized pixel interpolation unit 310 for generating a non-polarized image, a polarized pixel averaging processing unit 311 for generating a high-resolution polarized image using the non-polarized image, a non-polarized pixel averaging processing unit 312, a central pixel acquisition unit 313, and a correlation processing unit 314.

[0028] The non-polarized pixel interpolation section 310 generates a non-polarized image by calculating pixel values ​​at pixel positions where no non-polarized pixel value is obtained in the captured image acquired by the polarization imaging section 20, through an interpolation process using the pixel values ​​of surrounding non-polarized pixels. The non-polarized pixel interpolation section 310 outputs the generated non-polarized image to the non-polarized pixel averaging processing section 312 and the central pixel acquisition section 313.

[0029] The polarization pixel averaging processor 311 uses the pixel of interest and the surrounding pixels located around the pixel of interest to calculate a pixel average value for each polarization direction and outputs the calculated average value to the correlation processor 314 .

[0030] The non-polarized pixel averaging processor 312 uses the pixel of interest and the surrounding pixels located around the pixel of interest to calculate pixel average values ​​from pixels at the same pixel positions as when the polarized pixel averaging processor 311 calculates the pixel average values ​​for each polarization direction, and outputs the calculated pixel average values ​​to the correlation processor 314.

[0031] The central pixel acquisition unit 313 extracts the pixel value of the pixel of interest from the unpolarized image and outputs it to the correlation processing unit 314 .

[0032] The correlation processing unit 314 calculates a pixel value for each polarization direction of the pixel of interest from the pixel average value for each polarization direction calculated by the polarization pixel averaging processing unit 311, the pixel average value calculated by the non-polarized pixel averaging processing unit 312 corresponding to the pixel average value for each polarization direction, and the pixel value of the pixel of interest extracted by the central pixel acquisition unit 313.

[0033] The demosaic process of the polarized pixels will be described with reference to FIG. 6. FIG. 6(a) shows a polarized image, and FIG. 6(b) shows a non-polarized image. The polarized pixel averaging processor 311 calculates the pixel average value for each polarization direction using pixels at the pixel of interest position "x,y" in the polarized image and the surrounding pixel positions "x-1,y-1", "x,y-1", "x+1,y-1", "x-1,y", "x+1,y", "x-1,y+1", "x,y+1", and "x+1,y+1". Here, the pixel value at the pixel of interest position in the polarized image is "P(x,y)", and the pixel values ​​at the surrounding pixel positions are "P(x-1,y-1)", "P(x,y-1)", "P(x+1,y-1)", "P(x-1,y)", "P(x+1,y)", "P(x-1,y+1)", "P(x,y+1)", and "P(x+1,y+1)". In addition, the pixel value of the pixel of interest in the unpolarized image is "Q(x,y)", and the pixel values ​​of the surrounding pixels are "Q(x-1,y-1)", "Q(x,y-1)", "Q(x+1,y-1)", "Q(x-1,y)", "Q(x+1,y)", "Q(x-1,y+1)", "Q(x,y+1)", and "Q(x+1,y+1)".

[0034] The polarization pixel averaging processor 311 calculates a pixel average value mP0(x,y) at the pixel of interest position "x,y" where the polarization direction is "α0=90°" based on, for example, equation (1). Similarly, the polarization pixel averaging processor 311 calculates a pixel average value mP1(x,y) at the pixel of interest position "x,y" where the polarization direction is "α1=0°" based on equation (2). Furthermore, the polarization pixel averaging processor 311 calculates a pixel average value mP2(x,y) at the pixel of interest position "x,y" where the polarization direction is "α2=135°" based on equation (3). mP0(x,y) = P(x,y) (1) mP1(x,y)=(P(x-1,y-1)+P(x+1,y-1)+P(x-1,y+1) +P(x+1,y+1)) / 4 (2) mP2(x,y)=(P(x,y-1)+P(x,y+1)) / 2 ···(3)

[0035] The non-polarized pixel averaging processor 312 calculates pixel average values ​​from pixel values ​​at pixel positions equivalent to those calculated for each polarization direction, using a pixel of interest and pixels surrounding the pixel of interest in the non-polarized image. For example, the non-polarized pixel averaging processor 312 calculates a pixel average value mQ0(x,y) for the pixel average value whose polarization direction is "α0=0°" at the pixel of interest position "x,y" based on equation (4). Similarly, the non-polarized pixel averaging processor 312 calculates a pixel average value mQ1(x,y) for the pixel average value whose polarization direction is "α1=0°" based on equation (5). Moreover, the non-polarized pixel averaging processor 312 calculates a pixel average value mQ2(x,y) for the pixel average value whose polarization direction is "α2=135°" based on equation (6). mQ0(x,y) = Q(x,y) (4) mQ1(x,y)=(Q(x-1,y-1)+Q(x+1,y-1)+Q(x-1,y+1) +Q(x+1,y+1)) / 4 (5) mQ2(x,y)=(Q(x,y-1)+Q(x,y+1)) / 2 ···(6)

[0036] The correlation processing unit 314 assumes that there is a positive correlation between polarized pixels and non-polarized pixels, and calculates pixel values ​​for each polarization direction at the position of the pixel of interest from the pixel average value calculated from the polarized image, the pixel average value calculated from the non-polarized image, and the pixel value of the pixel of interest in the non-polarized image.

[0037] For example, the correlation processing unit 314 calculates a pixel value P0(x,y) of a polarized pixel at the pixel of interest position "x,y" with a polarization direction of "α0=90°" based on equation (7). Similarly, the correlation processing unit 314 calculates a pixel value P1(x,y) of a polarized pixel at the pixel of interest position "x,y" with a polarization direction of "α1=0°" based on equation (8). Furthermore, the correlation processing unit 314 calculates a pixel value P2(x,y) of a polarized pixel at the pixel of interest position "x,y" with a polarization direction of "α2=135°" based on equation (9). The correlation processing unit 314 outputs a pixel value Q(x,y) as the pixel value of the pixel of interest of the unpolarized image. P0(x,y)=mP0(x,y)·Q(x,y) / mQ0(x,y)···(7) P1(x,y)=mP1(x,y)·Q(x,y) / mQ1(x,y)···(8) P2(x,y)=mP2(x,y)·Q(x,y) / mQ2(x,y)···(9)

[0038] The demosaic processor 31 performs the above-mentioned process with each pixel of the polarized image as a pixel of interest, thereby generating a polarized image for each polarization direction. In addition, the demosaic processor 31 normalizes the signal components of the non-polarized pixels by the pixel average value and superimposes them on the polarized pixels, so that it is possible to improve problems such as aliasing caused by the frequency limit of the polarized pixels, and generate a polarized image with the same resolution as the non-polarized image for each polarization direction. If the pixel array corresponds to a Bayer array, a non-polarized image and a polarized image for each polarization direction may be generated by performing a process similar to that disclosed in, for example, "B. Gunturk, J. Glotzbach, Y. Altunbasak, R. schafer, and R. Mersereau, "Demosaicing: Color filter array interpolation,".in IEEE Signal Processing Magazine, vol. 22, no. 1, Jan 2005." In addition, the demosaic processor 31 is only required to generate a non-polarized image and a polarized image for each polarization direction, and the demosaic process is not limited to the above-mentioned method.

[0039] The invalid pixel detection unit 32 detects invalid pixels by comparing the pixel values ​​of the non-polarized pixels and the polarized pixels with a preset threshold value. As the threshold value, a saturation detection threshold value Tha for detecting saturated pixels and a black crush detection threshold value Thb for detecting black crush pixels are set. The invalid pixel detection unit 32 compares the pixel value of each pixel with the saturation detection threshold value Tha, and determines a pixel having a pixel value greater than the saturation detection threshold value Tha as an invalid pixel. The invalid pixel detection unit 32 also compares the pixel value of each pixel with the black crush detection threshold value Thb, and determines a pixel having a pixel value smaller than the black crush detection threshold value Thb as an invalid pixel. Furthermore, the non-polarized pixels do not have a polarizing filter and therefore have a larger amount of incident light (higher sensitivity) than the polarized pixels, while the polarized pixels have a smaller amount of incident light (lower sensitivity) than the polarized pixels because the amount of light is reduced by the polarizing filter. Therefore, the pixel value of the non-polarized image may be compared with the saturation detection threshold value Tha to detect invalid pixels, and the pixel value of the polarized image may be compared with the black crush detection threshold value Thb to detect invalid pixels.

[0040] The saturation detection threshold Tha and the black crush detection threshold Thb may be set to have the same level difference between the specified level and the threshold, for example, the level difference between the maximum output level of the polarization imaging section 20 and the saturation detection threshold Tha, and the level difference between the minimum output level of the polarization imaging section 20 and the black crush detection threshold Thb, or the thresholds may be set individually according to the characteristics of the polarization imaging section 20. For example, it is known that the image sensor 21 generates shot noise depending on the amount of signal charge, and the shot noise is proportional to the square root of the amount of signal charge. Therefore, the black crush detection threshold Thb is set to be higher than the minimum output value of the polarization imaging section 20 by a first level, and the saturation detection threshold Tha is set to be lower than the maximum output value of the polarization imaging section 20 by a second level that is lower than the first level. By setting the saturation detection threshold Tha and the black crush detection threshold Thb in this way, it is possible to prevent the influence of noise when detecting an invalid pixel that has caused black crush from becoming greater than when detecting an invalid pixel that has caused saturation. In addition, the saturation detection threshold Tha and the black crush detection threshold Thb may be set according to the optical characteristics of the polarizing filter 22. For example, when the transmittance of the polarizing filter 22 is low, the amount of light incident on the image sensor 21 is small, and the signal level of the pixel signal is therefore small. In addition, when adjusting the level of the pixel signal, the gain becomes high, and there is a risk that the effect of noise will be greater than when the transmittance is high. Therefore, the level difference between the specified level and the threshold may be greater when the transmittance of the polarizing filter 22 is low than when the transmittance is high.

[0041] The invalid pixel detection unit 32 compares the saturation detection threshold Tha, the black crush detection threshold Thb (<Tha), and the pixel value I(x,θ) of pixel x. When I(x,θ)>Tha or Thb>I(x,θ), pixel x is set as an invalid pixel. Here, "x" is the index of the pixel. Also, "θ" indicates the type of pixel. For example, when the polarization direction is 0°, "θ = 0"; when it is 90°, "θ = 90"; when it is 135°, "θ = 135"; and when it is an unpolarized pixel, "θ = -1". The invalid pixel detection unit 32 sets an invalid flag V(x,θ) for each pixel. The invalid flag of the pixel determined to be an invalid pixel is V(x,θ)=1, and the invalid flag of the pixel not determined to be an invalid pixel is V(x,θ)=0. The invalid pixel detection unit 32 outputs the invalid flag V(x,θ) indicating the detection result of the invalid pixel to the polarization information generation unit 33.

[0042] The polarization information generation unit 33 performs polarization information generation processing based on the unpolarized image generated by the demosaicing processing unit 31 and the polarization images for each polarization direction, and switches the polarization information generation processing according to the detection result of the invalid pixels in the invalid pixel detection unit 32.

[0043] It is known that the polarization state of the subject can be expressed, for example, as a polarization model formula shown in Equation (10). In Equation (10), the parameters S0, S1, and S2 are Stokes parameters. The Stokes parameter S0 is the sum of the observed luminance of 0° polarization and the observed luminance of 90° polarization, and is a parameter indicating intensity. The Stokes parameter S1 is the difference between the observed luminance of 0° polarization and the observed luminance of 90° polarization, and the Stokes parameter S2 is a parameter indicating the difference between the observed luminance of 45° polarization and the observed luminance of 135° polarization.

[0044]

Equation

[0045] The polarization information generating unit 33 performs fitting to the polarization model equation shown in equation (10) using pixel values ​​of the unpolarized image and the polarized image for each polarization direction to calculate the Stokes parameters, and generates and outputs polarization information indicating the calculated Stokes parameters. In fitting to the polarization model equation, the polarization information generating unit 33 switches the polarization information generation process, i.e., the Stokes parameter calculation method, based on the invalid flag V(x, θ) from the invalid pixel detecting unit 32, and calculates the Stokes parameters S0, S1, S2 in the polarization model equation of equation (10) by, for example, the least squares method, using the pixel values ​​of pixels that have not been determined to be invalid pixels.

[0046] For example, as shown in (a) of FIG. 4, a 2×2 pixel polarized pixel block is composed of three polarized pixels (pixels with polarization directions of "0°", "90°", and "135°") and one non-polarized pixel. FIG. 7 illustrates the relationship between the luminance of a non-polarized pixel, the luminance of a polarized pixel, and the angle of the polarization direction (part 1). FIG. 7 (a) illustrates a case in which all pixels in the polarized pixel block are not determined to be invalid pixels. When all pixels in the polarized pixel block are not determined to be invalid pixels, that is, when the invalid flag V(x,-1)=V(x,0)=V(x,90)=V(x,135)=0, the Stokes parameter S0 is calculated based on Equation (11). The coefficient K in Equation (11) is a coefficient that absorbs the sensitivity difference between the polarized pixel and the non-polarized pixel.

[0047]

number

[0048] Furthermore, the polarization information generating unit 33 calculates the Stokes parameters S2 and S1 by the least squares method or the like using the pixel values ​​I(x,0), I(x,90), and I(x,135) and the Stokes parameter S0.

[0049] Next, a case where only the non-polarized pixels in the polarization pixel block are determined to be invalid pixels will be described. FIG. 7B illustrates a case where only the non-polarized pixels are determined to be invalid pixels. When the amount of light incident on the polarization image capture unit 20 increases and saturation occurs in the non-polarized pixels, for example, the pixel value of the non-polarized pixels becomes a pixel value I(x,-1) that is a lower level than the pixel value Isa corresponding to the amount of incident light. The non-polarized pixels whose pixel values ​​are saturated in this manner are determined to be invalid pixels. Therefore, when only the non-polarized pixels are determined to be invalid pixels, that is, when the invalid flags V(x,-1)=1 and V(x,0)=V(x,90)=V(x,135)=0, the polarization information generation unit 33 calculates the Stokes parameters S2, S1, and S0 from the pixel values ​​I(x,0), I(x,90), and I(x,135) of the polarization pixels that are not invalid pixels and the polarization model formula shown in formula (10) by the least squares method or the like.

[0050] Next, a case where only one polarized pixel in a polarized pixel block is determined to be an invalid pixel will be described. FIG. 7(c) illustrates an example where only one polarized pixel in a polarized pixel block is determined to be an invalid pixel. When the amount of light incident on the polarized image capturing unit 20 decreases and, for example, a polarized pixel with a polarization direction of 90 degrees is blacked out, the pixel value of this polarized pixel becomes a pixel value I(x,90) that is a higher level than the pixel value Isb corresponding to the amount of incident light. A polarized pixel whose pixel value is blacked out in this manner is determined to be an invalid pixel. FIG. 7(c) illustrates an example where the average value I(x,avg) of the polarization model changes to an average value Ibc due to blacked out of the polarized pixel. Therefore, when only one polarized pixel in a polarized pixel block is determined to be an invalid pixel, for example, when the invalid flag V(x,-1)=V(x,0)=V(x,135),V(x,90)=1, the polarization information generating unit 33 calculates the Stokes parameters without using the 135° polarized pixel determined to be an invalid pixel. The polarization information generating unit 33 calculates the Stokes parameter S0 based on the formula (12). Note that the coefficient K is a coefficient that absorbs the difference in sensitivity between the polarization pixels and the non-polarization pixels. S0 = K × I(x,-1) (12)

[0051] Furthermore, the polarization information generating unit 33 calculates the Stokes parameters S2 and S1 by the least squares method or the like using the pixel values ​​I(x,0), I(x,90) and the Stokes parameter S0.

[0052] If two of the three polarization pixels in the polarization pixel block are invalid pixels, the polarization information generating unit 33 cannot calculate the Stokes parameters S2, S1, and S0. In this case, the polarization information generating unit 33 may use information indicating that the polarization characteristics cannot be acquired as the polarization information.

[0053] Also, as shown in (b) of FIG. 4, a polarized pixel block of 2×2 pixels is composed of two polarized pixels (pixels with polarization directions of “90°” and “135°”) and two non-polarized pixels. FIG. 8 illustrates the relationship between the luminance of a non-polarized pixel, the luminance of a polarized pixel, and the angle of the polarization direction (part 2). FIG. 8 (a) illustrates a case in which all pixels in the polarized pixel block are not determined to be invalid pixels. When all pixels in the polarized pixel block are not determined to be invalid pixels, that is, when the invalid flag V(x,-1a)=V(x,-1b)=V(x,90)=V(x,135)=0, the Stokes parameter S0 is calculated based on the formula (11). Note that the invalid flag V(x,-1a) indicates the detection result of one of the two non-polarized pixels, and the invalid flag V(x,-1b) indicates the detection result of the other of the two non-polarized pixels.

[0054] Furthermore, the polarization information generating unit 33 calculates the Stokes parameters S2 and S1 by the least squares method or the like using the pixel values ​​I(x, 90), I(x, 135) and the Stokes parameter S0.

[0055] Next, a case where only one non-polarized pixel in a polarization pixel block is determined to be an invalid pixel will be described. FIG. 8B illustrates a case where only one non-polarized pixel is determined to be an invalid pixel. When the amount of incident light to the polarization imaging unit 20 increases and saturation occurs in the non-polarized pixel, for example, the pixel value of the non-polarized pixel becomes a pixel value I(x, -1a) that is a lower level than the pixel value Isa corresponding to the amount of incident light. The non-polarized pixel whose pixel value is saturated in this way is determined to be an invalid pixel. Therefore, when only one non-polarized pixel is determined to be an invalid pixel, the polarization information generating unit 33 sets the invalid flag V(x, -1a)=1, V(x, -1b)=V(x, 90)=V(x, 135)=0. In this case, the pixels used to calculate the Stokes parameters are one non-polarized pixel and two polarized pixels, similar to the case where only one polarized pixel in a polarization pixel block is determined to be an invalid pixel in the pixel configuration shown in FIG. 4A. Therefore, the polarization information generating section 33 calculates the Stokes parameters S2, S1, and S0 as described above, using the pixel values ​​of the non-polarization pixels that are not invalid pixels and the two polarization pixels.

[0056] In addition, when two non-polarized pixels or one polarized pixel in the polarized pixel block are determined to be invalid pixels, the polarization information generating unit 33 cannot calculate the Stokes parameters S2, S1, and S0. FIG. 8(c) illustrates a case where one polarized pixel in the polarized pixel block is determined to be an invalid pixel. When the amount of incident light to the polarization imaging unit 20 decreases and, for example, a polarized pixel with a polarization direction of 90 degrees is crushed, the pixel value of this polarized pixel becomes a pixel value I(x,90) that is a higher level than the pixel value Isb corresponding to the amount of incident light. In this way, the polarized pixel whose pixel value is crushed is determined to be an invalid pixel. In addition, FIG. 8(c) illustrates an example in which the average value I(x,avg) of the polarization model changes to the average value Ibc due to the crushed black of the polarized pixel. Therefore, when only one polarization pixel in a polarization pixel block is determined to be an invalid pixel, the polarization information generating unit 33 sets the invalid flags V(x, 90) = 1, V(x, -1a) = V(x, -1b) = V(x, 135) = 0. Here, since the polarization pixel with a polarization direction of 90 degrees is an invalid pixel, fitting to the polarization model formula cannot be performed when the pixel values ​​of the polarization pixel with a polarization direction of 135 degrees and two non-polarized pixels are used. Therefore, the polarization information generating unit 33 may use information indicating that the polarization characteristics cannot be acquired as the polarization information.

[0057] 9 is a flowchart showing the operation of the image processing unit. In step ST1, the image processing unit acquires a captured image. The image processing unit generates the captured image using a polarization imaging unit composed of polarization pixels and non-polarization pixels, and proceeds to step ST2.

[0058] In step ST2, the image processing unit generates a polarized image and a non-polarized image for each polarization direction. The image processing unit performs demosaic processing using the captured image to generate a polarized image and a non-polarized image for each polarization direction, and then proceeds to step ST3.

[0059] In step ST3, the image processing unit detects invalid pixels. The image processing unit compares the pixel values ​​of the non-polarized pixels and the polarized pixels with a preset saturation detection threshold Tha and a preset black crush detection threshold Thb, detects pixels whose pixel values ​​are greater than the saturation detection threshold Tha and pixels whose pixel values ​​are smaller than the black crush detection threshold Thb as invalid pixels, and proceeds to step ST4.

[0060] In step ST4, the image processing unit generates polarization information. Based on the pixel values ​​for each polarization direction, the pixel values ​​of non-polarized pixels, and the invalid pixel detection result, the image processing unit performs fitting to a polarization model formula using the pixel values ​​of polarized pixels and non-polarized pixels that are not invalid pixels, and calculates, for example, Stokes parameters. The image processing unit generates polarization information indicating the calculated Stokes parameters.

[0061] As described above, the image processing unit generates a non-polarized image and a polarized image for each polarization direction in the demosaic processing unit 31 from the captured image generated by the polarization imaging unit 20 in which non-polarized pixels and at least two polarized pixels for each polarization direction are arranged. The image processing unit also generates polarization information indicating the polarization characteristics of the subject included in the captured image in the polarization information generating unit 33 from the non-polarized image and the polarized image generated by the demosaic processing unit 31. In this way, since polarization information is generated using not only the polarized image but also the highly sensitive non-polarized image, it becomes possible to obtain polarization information with higher accuracy than when polarization information is generated based on the polarized image. Furthermore, the image processing unit detects invalid pixels from the non-polarized image and the polarized image, and generates polarization information without using invalid pixels, so that it becomes possible to obtain correct polarization information.

[0062] <2. Other embodiments> In the above-mentioned image processing unit, the polarization angles of the polarization pixels are 0°, 90°, and 135°, but the polarization angles may be configured so as not to be orthogonal. For example, the polarization pixel block may be configured with one non-polarizing pixel and three polarization pixels with polarization angles of 0°, 60°, and 120°.

[0063] By setting the polarization angle in this way, even if any of the polarization pixels with three different polarization directions becomes a crushed pixel, the remaining two polarization directions are not orthogonal, so fitting to the polarization model formula is possible based on the non-polarized pixel and the two polarization pixels that are not orthogonal. Therefore, even if any of the polarization pixels with three different polarization directions becomes a crushed pixel, polarization information can be obtained.

[0064] In the above embodiment, the polarization information generating unit 33 calculates the Stokes parameters as the polarization information, but the polarization information is not limited to the case where it indicates the Stokes parameters. For example, the polarization information generating unit 33 may generate normal information as the polarization information. In this case, the polarization information generating unit 33 calculates the azimuth angle φ based on, for example, equation (13). The polarization information generating unit 33 also calculates the zenith angle θ based on equation (14). Note that the parameters A, B, C, and D in equation (14) are values ​​calculated by equations (15) to (18), and the degree of polarization ρ shown in equations (15) and (16) is calculated based on equation (19). In addition, in equations (15), (17), and (18), the parameters A, B, C, and D are calculated using the refractive index n of the object OB.

[0065]

number

[0066] Furthermore, the polarization imaging section 20 may be configured to generate a color captured image, and the image processing section 30 may generate polarization information based on the color captured image. Fig. 10 illustrates an example of the configuration of a polarization imaging section that generates a color captured image. When a color captured image is generated by the polarization imaging section 20, a color mosaic filter 23 is provided on the incident surface of the imaging element 21. The color mosaic filter 23 is not limited to being provided between the imaging element 21 and the polarizing filter 22, and may also be provided on the incident surface of the polarizing filter 22.

[0067] The polarizing filter 22 and the color mosaic filter 23 are configured to have pixels of the same color for each polarization direction so that there is no influence due to color differences between polarizing pixels with different polarization directions. Also, the polarizing filter 22 and the color mosaic filter 23 are configured so that the pixel values ​​of the non-polarized pixels can be obtained for each color.

[0068] FIG. 11 illustrates the relationship between a color mosaic filter and a polarizing filter (two polarization directions). In FIG. 11(a), the polarizing filter 22 is configured to receive two polarizing pixels and two non-polarizing pixels with different polarization directions in a 2×2 pixel region. The color mosaic filter 23 is configured with the 2×2 pixel region as color units of red (R), green (G), and blue (B), and the 4×4 pixel region is configured with a red 2×2 pixel region, a blue 2×2 pixel region, and two green 2×2 pixel regions. In this case, the polarization imaging unit 20 generates pixel values ​​of polarizing pixels for each of the two polarization directions and pixel values ​​of non-polarizing pixels of any of the colors red, green, and blue for each 2×2 pixel region.

[0069] In (b) of Fig. 11, the color mosaic filter 23 has a 2x2 pixel region divided into color units of red (R), green (G), and blue (B). The polarizing filter 22 has a green 2x2 pixel region with two polarizing pixels and two non-polarizing pixels having different polarization directions, and the 4x4 pixel region has a red 2x2 pixel region, a blue 2x2 pixel region, and two green 2x2 pixel regions. In this case, the polarization imaging section 20 generates, for each 2x2 pixel region, a pixel value of a non-polarized pixel of any one of red, green, or blue, and a pixel value of a polarized pixel of two polarization directions for green.

[0070] In (c) of Fig. 11, the polarizing filter 22 is configured to receive two polarizing pixels and two non-polarizing pixels with different polarization directions in a 2x2 pixel region. The color mosaic filter 23 is configured to have a 2x2 pixel region with three green (G) pixels and one red (R) or blue (B) pixel, and the 4x4 pixel region is configured with two 2x2 pixel regions including a red pixel and two 2x2 pixel regions including a blue pixel. In this case, the polarization imaging section 20 generates pixel values ​​of non-polarizing pixels, which are a green pixel and a red pixel or a green pixel and a blue pixel, and pixel values ​​of polarizing pixels for each of the two polarization directions in green, for each 2x2 pixel region.

[0071] In FIG. 11(d), the color mosaic filter 23 is configured such that a 2×2 pixel region is composed of two white (W) pixels and two pixels of any color of red (R), green (G), or blue (B). The 4×4 pixel region is composed of a 2×2 pixel region including a red pixel, a 2×2 pixel region including a blue pixel, and two 2×2 pixel regions including a green pixel. The polarizing filter 22 is configured such that the white pixel in the 2×2 pixel region including a green pixel is a polarizing pixel having the same polarization direction, and two polarizing pixels of two polarization directions are provided in each of the 4×4 pixel regions. In this case, the polarization image capture unit 20 generates a pixel value of a non-polarized pixel of any color of red, green, or blue for each 2×2 pixel region, and a pixel value of a polarizing pixel of any of the two polarization directions in the 2×2 pixel region including a green pixel.

[0072] In FIG. 11(e), the color mosaic filter 23 is configured such that a 2×2 pixel region is composed of two white (W) pixels, a green (G) pixel, and a red (R) pixel, or two white (W) pixels, a green (G) pixel, and a blue (B) pixel. The 4×4 pixel region is composed of two 2×2 pixel regions including a red pixel and two 2×2 pixel regions including a blue pixel. The polarizing filter 22 is configured such that one white pixel in the 2×2 pixel region is a polarizing pixel, and the 4×4 pixel region is provided with two polarizing pixels each having two polarization directions. In this case, the polarization image capture unit 20 generates pixel values ​​of green and red or green and blue non-polarized pixels and pixel values ​​of polarized pixels having one of the two polarization directions for each 2×2 pixel region.

[0073] Also, in FIG. 11, the polarization image capture section 20 generates pixel values ​​of non-polarized pixels of each color and pixel values ​​of polarized pixels for each polarization direction for each 4×4 pixel region.

[0074] FIG. 12 illustrates the relationship between the color mosaic filter and the polarizing filter (three polarization directions). In FIG. 12(a), the color mosaic filter 23 is configured such that a 2×2 pixel region is made up of two white (W) pixels and two pixels of any color of red (R), green (G), or blue (B). The 4×4 pixel region is made up of a 2×2 pixel region including a red pixel, a 2×2 pixel region including a blue pixel, and two 2×2 pixel regions including a green pixel. The polarizing filter 22 is configured such that the white pixel in the 2×2 pixel region is a polarizing pixel of any polarization direction, and two polarizing pixels of each of the four polarization directions are provided in the 4×4 pixel region. In this case, the polarization imaging unit 20 generates a pixel value of a non-polarized pixel of any color of red, green, or blue, and a pixel value of a polarizing pixel of any polarization direction among the four polarization directions, for each 2×2 pixel region.

[0075] In FIG. 12B, the color mosaic filter 23 is configured such that a 2×2 pixel region is composed of two white (W) pixels, a green (G) pixel, and a red (R) pixel, or two white (W) pixels, a green (G) pixel, and a blue (B) pixel. The 4×4 pixel region is composed of two 2×2 pixel regions including a red pixel and two 2×2 pixel regions including a blue pixel. The polarizing filter 22 is configured such that the two white pixels in the 2×2 pixel region are polarized pixels having different polarization directions, and two polarized pixels having four polarization directions are provided in each of the 4×4 pixel regions. In this case, the polarization image capturing section 20 generates pixel values ​​of green and red or green and blue non-polarized pixels and pixel values ​​of polarized pixels having two polarization directions for each 2×2 pixel region. In this case, the polarization image capturing section 20 generates pixel values ​​of green and red or green and blue non-polarized pixels and pixel values ​​of polarized pixels having any two polarization directions among the four polarization directions for each 2×2 pixel region.

[0076] 12(c), the color mosaic filter 23 has a 2×2 pixel region as a red (R), green (G), and blue (B) color unit, and the 4×4 pixel region has a red 2×2 pixel region, a blue 2×2 pixel region, and two green 2×2 pixel regions. The polarizing filter 22 has four polarizing pixels with different polarization directions in the green 2×2 pixel region. In this case, the polarization image capture section 20 generates pixel values ​​of non-polarized pixels in the red or blue 2×2 pixel region, and generates pixel values ​​of polarized pixels of each polarization direction in the green 2×2 pixel region.

[0077] In addition, in (a) and (b) of FIG. 12, the polarization image capture section 20 generates pixel values ​​of non-polarized pixels of each color and pixel values ​​of polarized pixels for each polarization direction for each 4×4 pixel region.

[0078] When a color captured image is generated by the polarization imaging unit 20, the demosaic processing unit 31 generates a non-polarized image for each color component from the color image. The demosaic processing unit 31 also generates a polarized image for each polarization direction. The polarization information generating unit 33 generates polarization information using the polarized image and the non-polarized image generated by the demosaic processing unit 31. Since the polarized pixels are white or pixels of the same color as described above, there is no influence due to color differences between the polarized pixels. Therefore, polarization information can be generated correctly even when a color captured image is used. In addition, since the green pixels are polarized pixels in (c) of FIG. 12, the sensitivity of the green pixels is reduced compared to the red and blue pixels, which are non-polarized pixels. Therefore, if the pixel value of the green pixels is corrected according to the reduced sensitivity of the polarized pixels, a color component image similar to that in the case where the pixels of each color component are non-polarized pixels can be generated by performing demosaic processing of the conventional Bayer array.

[0079] In this way, if a color image is generated by the polarization imaging section, it becomes possible to generate accurate polarization information by using color. Also, when a color image is generated by the polarization imaging section, the image processing section 30 can generate a non-polarized image for each color by performing integration processing using pixels of the same color.

[0080] Furthermore, the configuration of the image processing section is not limited to the configuration shown in Fig. 1. For example, since the image processing section 30 is configured to correspond to the pixel configuration of the polarization imaging section 20 as described above, the polarization imaging section 20 and the image processing section 30 may be configured as an integrated unit.

[0081] <3. Application Examples> The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, a robot, a construction machine, or an agricultural machine (tractor). If applied to such fields, accurate polarization information can be acquired, so that, for example, the surrounding environment can be accurately grasped in three dimensions based on the polarization information, thereby reducing the fatigue of the driver or worker. In addition, automatic driving and the like can be performed more safely.

[0082] The technology disclosed herein can also be applied to the medical field. For example, if applied to the case where captured images of the surgical site are used during surgery, it becomes possible to obtain a three-dimensional shape of the surgical site and an image without reflections with high accuracy, reducing the fatigue of the surgeon and enabling the surgery to be performed safely and more reliably.

[0083] The technology according to the present disclosure can also be applied to the field of public services, etc. For example, when an image of a subject is published in a book, magazine, etc., it becomes possible to accurately remove unnecessary reflection components, etc. from the image of the subject.

[0084] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed. Alternatively, the program can be installed in a general-purpose computer capable of executing various processes and executed.

[0085] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called package software.

[0086] In addition to being installed in a computer from a removable recording medium, the program may be transferred wirelessly or by wire from a download site to a computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it in a built-in recording medium such as a hard disk.

[0087] In addition, the effects described in this specification are merely examples and are not limited, and there may be additional effects that are not described. In addition, the present technology should not be interpreted as being limited to the above-mentioned embodiment. The embodiment of this technology discloses the present technology in the form of an example, and it is self-evident that a person skilled in the art can modify or substitute the embodiment without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration in order to determine the gist of the present technology.

[0088] Moreover, the image processing device of the present technology can also have the following configuration. (1) an invalid pixel detection unit that detects saturated pixels and crushed black pixels as invalid pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using the polarization imaging unit; a polarization information generating unit that performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating the polarization information depending on a detection result of the invalid pixel by the invalid pixel detecting unit; An image processing device comprising: (2) The image processing device according to (1), wherein the polarization information generating unit generates the polarization information using pixels of the plurality of polarization images when a pixel of the unpolarized image is detected as an invalid pixel. (3) The image processing device described in (1), wherein when a pixel of any of the plurality of polarized images is detected as an invalid pixel, the polarization information generation unit generates the polarization information using the pixel of the unpolarized image and the pixel of another polarized image that is not detected as an invalid pixel. (4) The plurality of polarized images are three or more polarized images with different polarization directions, The image processing device according to (3), wherein the polarization information generating section generates the polarization information using pixels of the unpolarized image and pixels of at least two or more polarized images that are not detected as invalid pixels. (5) The image processing device according to (4), wherein the at least two polarized images are polarized images whose polarization directions are not orthogonal to each other. (6) An image processing device according to any one of (1) to (5), wherein the polarization information generation unit generates the polarization information using pixels of the unpolarized image and pixels of the polarized image when no invalid pixels are detected. (7) The image processing device according to any one of (1) to (6), wherein the invalid pixel detection unit detects, as the invalid pixels, the saturated pixels from the unpolarized image and the crushed black pixels from the polarized image. (8) An image processing device according to any one of (1) to (7), wherein the invalid pixel detection unit detects pixels whose pixel values ​​are greater than a preset saturation detection threshold as the saturated pixels, and detects pixels whose pixel values ​​are smaller than a preset black crush detection threshold as the black crush pixels. (9) The image processing device according to (8), wherein the saturation detection threshold and the black crush detection threshold are set in accordance with characteristics of an image acquired by capturing the image using the polarization image capturing unit. (10) An image processing device as described in (9), wherein the black crush detection threshold is set to be higher than the minimum output value of the polarization imaging unit by a first level, and the saturation detection threshold is set to be lower than the maximum output value of the polarization imaging unit by a second level lower than the first level. [Explanation of symbols]

[0089] 10. System 20 Polarization imaging section 21 Imaging element 22 Polarizing filter 23. Color Mosaic Filter 30...Image processing unit 31 Demosaic processing unit 32 Invalid pixel detection unit 33...Polarization information generation section 211 Pixel array section 212 Vertical scanning circuit 213 Horizontal scanning circuit 310 Non-polarized pixel interpolation unit 311 Polarization pixel averaging processing unit 312 Non-polarized pixel averaging processing unit 313 Center pixel acquisition unit 314 Correlation processing unit

Claims

1. an invalid pixel detection unit that detects saturated pixels and crushed black pixels as invalid pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using the polarization image capturing unit; a polarization information generating unit that performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating the polarization information depending on a detection result of the invalid pixel by the invalid pixel detecting unit; An image processing device comprising:

2. When a pixel of the unpolarized image is detected as an invalid pixel, the polarization information generating section generates the polarization information using pixels of the plurality of polarization images. The image processing device according to claim 1 .

3. When a pixel of any of the plurality of polarized images is detected as an invalid pixel, the polarization information generating unit generates the polarization information using the pixel of the unpolarized image and a pixel of another polarized image that is not detected as an invalid pixel. The image processing device according to claim 1 .

4. the plurality of polarized images are three or more polarized images with different polarization directions, The polarization information generating unit generates the polarization information using pixels of the unpolarized image and pixels of at least two or more polarized images that are not detected as invalid pixels. The image processing device according to claim 3 .

5. The at least two polarized images are polarized images whose polarization directions are not perpendicular to each other. The image processing device according to claim 4.

6. When the invalid pixel is not detected, the polarization information generating unit generates the polarization information using the pixels of the unpolarized image and the pixels of the polarized image. The image processing device according to claim 1 .

7. The invalid pixel detection unit detects the saturated pixels from the unpolarized image and the black crushed pixels from the polarized image as the invalid pixels. The image processing device according to claim 1 .

8. The invalid pixel detection unit detects pixels having pixel values ​​larger than a preset saturation detection threshold as the saturated pixels, and detects pixels having pixel values ​​smaller than a preset black crush detection threshold as the black crush pixels. The image processing device according to claim 1 .

9. The saturation detection threshold and the black crush detection threshold are set according to the characteristics of an image acquired by performing the imaging with the polarization imaging unit. The image processing device according to claim 8.

10. The black crush detection threshold is set to be higher than the minimum output value of the polarization image capturing unit by a first level, and the saturation detection threshold is set to be lower than the maximum output value of the polarization image capturing unit by a second level lower than the first level. The image processing device according to claim 9 .

11. detecting, as invalid pixels, saturated pixels and crushed black pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using the polarization image capturing unit, using an invalid pixel detection unit; A polarization information generating unit performs a process of generating polarization information based on the unpolarized image and the polarized image, and switches the process of generating the polarization information depending on a detection result of the invalid pixels by the invalid pixel detecting unit. An image processing method comprising:

12. A program for causing a computer to generate polarization information, a step of detecting saturated pixels and crushed black pixels as invalid pixels from a non-polarized image and a plurality of polarized images having different polarization directions obtained by capturing images using a polarization image capturing unit; generating polarization information based on the unpolarized image and the polarized image; a step of switching a generation process of the polarization information in response to a detection result of the invalid pixel; A program for causing the computer to execute the above.

Images