Processing device, imaging device, lens device, processing method, and program

Abstract

To provide a processing device capable of acquiring highly accurate polarization information.SOLUTION: The processing device has an acquisition unit for acquiring a plurality of pieces of interpolated polarization component information at a first pixel based on first to fourth polarization component information acquired from each of first to fourth pixels having primary sensitivity to each of different polarization components and a calculating unit for calculating the polarization information at the first pixel using the plurality of pieces of post-interpolated polarization component information. The calculating unit calculates the polarization information by assigning a greater weight to the information estimated to have a high accuracy among the plurality of pieces of post-interpolated polarization component information than to the information estimated to have a low accuracy.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing device that acquires polarization information from polarization component information. [Background technology]

[0002] Observing the polarization state of light reflected from a subject makes it possible to highlight and detect the features of the subject. For example, by attaching a polarizing filter to the front of the imaging lens and capturing an image of the subject, it is possible to obtain an image that highlights the texture of the subject, such as color and contrast, and emphasizes or reduces the glare of reflected light from the surface of water, etc. Furthermore, by obtaining information on polarization components with different polarization directions from the subject, it is also possible to detect edges and defects in the subject.

[0003] Patent Document 1 discloses a method of interpolating polarization component information from an imaging element in which four types of polarization filters are arranged, and calculating polarization information such as the degree of polarization and polarization direction from the obtained interpolated information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4965615 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the method of Patent Document 1, polarization component information with high interpolation accuracy and polarization component information with low interpolation accuracy are treated equally to calculate the polarization information, so the calculation accuracy of the polarization information decreases due to the polarization component information with low interpolation accuracy.

[0006] An object of the present invention is to provide a processing device capable of acquiring highly accurate polarization information. [Means for solving the problem]

[0007] A processing device according to one aspect of the present invention includes an acquisition unit that acquires a plurality of pieces of interpolated polarization component information at a first pixel based on first to fourth polarization component information acquired from first to fourth pixels that have primary sensitivities to different polarization components, respectively; and a calculation unit that calculates the polarization information at the first pixel using the plurality of pieces of interpolated polarization component information. By least squares method and a calculation unit for calculating the interpolated polarization component information, the calculation unit weighting the information estimated to be highly accurate among the plurality of pieces of interpolated polarization component information more heavily than the information estimated to be less accurate. , the polarization direction in which the light intensity of the incident light is maximum, the maximum light intensity of the incident light, and the minimum light intensity of the incident light. The polarization information is calculated. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a processing device capable of acquiring highly accurate polarization information. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a configuration of an imaging apparatus according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating a configuration of a polarization imaging element. [Figure 3] FIG. 10 is a diagram illustrating an example of a polarizing filter array. [Figure 4] FIG. 10 is a diagram showing the polarization state of incident light and the light intensity relative to the angle of incidence. [Figure 5] FIG. 10 is a diagram illustrating an example of calculation of polarization information. [Figure 6] FIG. 10 is a diagram illustrating an example of calculation of unweighted polarization information. [Figure 7] FIG. 10 is a diagram illustrating an example of calculation of weighted polarization information. [Figure 8] 2 is a diagram showing a polarization pixel array in the polarization imaging element of Example 1. FIG. [Figure 9] FIG. 10 is a diagram showing a polarization color pixel array in a polarization imaging element according to a second embodiment. [Figure 10] FIG. 10 is a diagram showing values ​​of non-polarized component information B in Example 3. [Figure 11] FIG. 10 is a diagram showing a polarization color pixel array in a polarization imaging element according to a fourth embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a neighboring region according to a fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to designate the same components, and redundant explanations will be omitted.

[0011] FIG. 1 is a diagram showing the configuration of an imaging device 1 according to an embodiment of the present invention. The imaging device 1 includes a photographing optical system 11, a polarization imaging element 12, an interpolation processing unit (acquisition unit) 13, a polarization information calculation unit (calculation unit) 14, and a recording unit 15. The photographing optical system 11 forms an optical image of a subject on the imaging surface of the polarization imaging element 12. While the imaging device 1 includes the photographing optical system 11 in this embodiment, it may be configured to allow an interchangeable lens to be detachably attached. Furthermore, the imaging device 1 does not need to include the interpolation processing unit 13, the polarization information calculation unit 14, and the recording unit 15; external devices having equivalent functions may be used instead. For example, the image information output from the interpolation processing unit 13 and the polarization information calculated by the polarization information calculation unit 14 may be recorded in an external recording device. Furthermore, demosaicing and calculation of polarization information may be performed by a processing device such as a personal computer (PC) having equivalent functions to the interpolation processing unit 13 and the polarization information calculation unit 14.

[0012] FIG. 2 is a diagram showing the configuration of the polarization imaging element 12. The polarization imaging element 12 has a pixel array 121 including a plurality of pixels arranged two-dimensionally in the horizontal and vertical directions on its imaging surface. Here, the horizontal and vertical directions are two directions that are perpendicular to each other. The polarization imaging element 12 can be a CCD-type element, a CMOS-type element, or the like. The polarization imaging element 12 also has a polarization filter array 122 including a plurality of polarization filters arranged two-dimensionally on the light-receiving surface side of the pixel array 121. As a result, the polarization imaging element 12 has a polarization pixel array with polarization filtering function. The polarization pixel array is a pixel array in which the pixel array 121 and the polarization filter array 122 are considered as a single unit.

[0013] Fig. 3 is a diagram showing an example of the polarization filter array 122. The x-axis and y-axis are two axes that are parallel to the pixel array 121 and perpendicular to each other. The directions of the x-axis and y-axis may be different from those shown in Fig. 3 as long as the above conditions are met. In Fig. 3, the polarization filter array 122 is a filter array in which 6 × 6 filters are two-dimensionally arranged in the x-axis direction (horizontal direction) and the y-axis direction (vertical direction).

[0014] Each polarizing filter in the polarizing filter array 122 is configured so that a specific polarized component has a higher transmittance than other polarized components. The polarizing filter may be a linear polarizing filter that has a high transmittance for polarized components of a specific polarization orientation, and the polarization orientation of the linear polarizing filter is the angle θ (0 degrees≦θ<180 degrees) between the polarization orientation of the polarized component with the highest transmittance and the x-axis. Alternatively, the polarizing filter may be a circular polarizing filter that has a high transmittance for either right-handed circularly polarized light or left-handed circularly polarized light.

[0015] In FIG. 3, the polarization filter array 122 is composed of polarization filters (hereinafter referred to as the θ1 filter, θ2 filter, θ3 filter, and θ4 filter) with four different polarization orientations θ1, θ2, θ3, and θ4 (denoted by 1, 2, 3, and 4 in the figure, respectively). The θ1 filter, θ2 filter, θ3 filter, and θ4 filter are each provided for one of the pixels included in the pixel array 121. As a result, multiple pixels provided with one of the θ1 filter, θ2 filter, θ3 filter, and θ4 filter each have primary sensitivity to polarization components with mutually different polarization orientations. In other words, the polarization pixel array has mutually different polarization sensitivity characteristics. Note that, although four filters are provided in this embodiment, five or more filters may be provided.

[0016] In the following description, of all pixels in the pixel array 121, pixels where a θ1 filter is arranged are referred to as "θ1 pixels." Similarly, pixels where a θ2 filter is arranged are referred to as "θ2 pixels," pixels where a θ3 filter is arranged are referred to as "θ3 pixels," and pixels where a θ4 filter is arranged are referred to as "θ4 pixels." Furthermore, pixels that can acquire each polarization component are referred to as polarization pixels.

[0017] The polarization filter array 122 may include regions where no polarization filters are provided or regions where filters that transmit polarization components of any polarization orientation are provided. The polarization components that can be acquired by the pixels (non-polarization pixels) corresponding to these regions are called non-polarization components.

[0018] The polarization imaging element 12 may be a polarization color imaging element equipped with a color filter array including color filters that have a higher transmittance for a specific wavelength component than for other wavelength components. An example of a color filter array is an array in which color filters that transmit the three primary colors of R (red), G (green), and B (blue) are arranged in a Bayer configuration. Instead of RGB color filters, complementary color filters of cyan (C), magenta (M), and yellow (Y) may be used, or infrared (IR) or ultraviolet (UV) color filters may be used. The pixels that can acquire the wavelength components of each color are called color pixels.

[0019] When the polarization imaging element 12 is a polarization color imaging element composed of multiple types of color pixel groups, the color pixel group with the highest ratio (pixel ratio) of pixels to the total number of pixels of the polarization imaging element 12 is designated as the first color pixel group. The pixel ratio of the first color pixel group is preferably 0.5 or more. In this case, information with high resolution can be acquired. Furthermore, the wavelength component to which the first color pixel group is primarily sensitive preferably includes a green wavelength component. Because the human eye is highly sensitive to green wavelength components, having more pixels that can acquire color information including green wavelength components can acquire information with high resolution. Note that pixels that can acquire color information including green wavelength components include G pixels, which are highly sensitive to green wavelength components, and W pixels, which are highly sensitive to wavelength components across the entire visible light range.

[0020] Furthermore, in the polarization pixel array in the embodiments described below, a square array pattern, such as a basic array pattern of 2x2 pixels or 8x8 pixels, is repeatedly arranged in the horizontal and vertical directions. Note that the basic array pattern is not limited to a square array pattern, and may be a rectangular array pattern of N1xN2 pixels (N1 and N2 are different natural numbers) or another polygonal array pattern. Furthermore, the polarization pixel array may be a random pixel array that does not have a basic array pattern.

[0021] The following describes the interpolation process for the information acquired by the polarization image sensor 12 and the polarization information calculation process. As described above, the subject image formed on the imaging surface of the polarization image sensor 12 is converted into charges corresponding to the intensity of incident light by each pixel of the polarization image sensor 12, and the charges are read out as electrical signals (pixel signals) from the polarization image sensor 12. The pixel signals read out from the polarization image sensor 12 are input to the interpolation processing unit 13 as luminance information for each pixel. The luminance information for each pixel includes polarization component information corresponding to the type of pixel.

[0022] Extracting luminance information from a specific type of pixel among multiple types of pixels results in a mosaic image consisting only of information on polarization components and wavelength components corresponding to the pixel type. The interpolation processing unit 13 performs demosaic (interpolation) processing on the mosaic image to calculate information not yet acquired at each pixel, and generates interpolated information (multiple pieces of interpolated polarization component information) for multiple polarization components at each pixel (first step). Note that the interpolation processing unit 13 may generate interpolated information for all polarization directions acquired by the polarization imaging element 12 at each pixel, or may generate interpolated information for some polarization directions. The interpolation processing unit 13 may also generate multiple pieces of interpolated information calculated using different interpolation methods for one polarization direction. The interpolation processing unit 13 may also use information other than the information to be calculated during the interpolation processing.

[0023] The interpolated information generated by the interpolation processing unit 13 is input to the polarization information calculation unit 14. The interpolated information may also be recorded in the recording unit 15. The polarization information calculation unit 14 calculates polarization information (α, Imax, Imin) that represents the polarization state of the incident light (second step). α is the polarization direction in which the light intensity is at its maximum, Imax is the maximum value of the light intensity, and Imin is the minimum value of the light intensity.

[0024] Here, we will explain how to calculate the polarization information (α, Imax, Imin). Consider the case where the light incident on the polarization imaging element 12 can be expressed as shown in FIG. 4. FIG. 4 shows the polarization state of the incident light and the light intensity versus the angle of incidence. The ellipse in FIG. 4(a) indicates the azimuth dependence of the amplitude of the incident light. The dashed lines in FIG. 4(a) represent the major and minor axes of the ellipse, and the polarization direction α is the angle between the major axis and the x-axis. The arrows in FIG. 4(a) represent the amplitude in the major and minor axis directions. The square of the amplitude is the light intensity, and FIG. 4(b) shows the luminance information corresponding to the light intensity of incident light that forms an angle θ with respect to the x-axis, i.e., the polarization component information I(θ) of the polarization direction θ of the incident light. Imax and Imin are the polarization component information corresponding to the polarization components in the major and minor axis directions, respectively. The polarization component information I(θ) is expressed by the following equation (1).

[0025]

number

[0026] Note that I(θ;α, Imax, Imin) is an expression of polarization component information I(θ) that clearly indicates polarization information (α, Imax, Imin).

[0027] According to equation (1), the polarization component information I(θ) changes in a 180-degree cycle and is determined by three coefficients (α, Imax, Imin). Therefore, in order to calculate the polarization information, it is necessary to obtain at least three pieces of polarization component information I(θ) when the angle θ is expressed as 0 degrees or more and less than 180 degrees. As an example, FIG. 5 shows three pieces of polarization component information, indicated by white circles, where θ = 0 degrees, 45 degrees, and 90 degrees, and the polarization component information I(θ) calculated from them.

[0028] On the other hand, as long as the above conditions are met, there are no particular restrictions on the polarization direction of the polarized components to be acquired. Therefore, by acquiring information I(θ) for any three or more polarized components, it is possible to obtain polarization information (α, Imax, Imin). Note that information on non-polarized components corresponds to the average value Iave of the light intensity of all polarized light.

[0029]

number

[0030] From equations (2) and (3), the light intensity I(θ+90 degrees) at a polarization direction of θ+90 degrees can be calculated from the average value Iave and the polarization component information I(θ), so non-polarized component information can also be considered a type of polarization component information. For example, if polarization component information for θ=0 degrees and 45 degrees and non-polarized component information are obtained, polarization component information for θ=90 degrees and θ=135 degrees can be calculated, so four polarization component information can be obtained.

[0031] An interpolation error occurs between the interpolated information for each pixel and the true value of the polarization component information for that pixel. Therefore, when calculating polarization information from four or more pieces of interpolated information, it is not possible to uniquely determine polarization information such that each polarization component information satisfies Equation (1). Therefore, when calculating polarization information from four or more pieces of interpolated information, the polarization information is calculated approximately using a method such as the least squares method.

[0032] To calculate polarization information from the first to Nth interpolation information by the least squares method, the polarization information (α, Imax, Imin) that minimizes the sum S of the following equation (4) is found.

[0033]

number

[0034] where D j is the j-th interpolation information, and φ j represents the polarization direction of the j-th interpolated information.

[0035] If there is no error in each piece of interpolated information, there exists polarization information (α, Imax, Imin) for which S=0, which coincides with the true polarization information (α', Imax', Imin').

[0036] FIG. 6 shows an example of calculation of unweighted polarization information. FIG. 6 shows four pieces of interpolated information for θ=0 degrees, 45 degrees, 90 degrees, and 135 degrees. In FIG. 6, the interpolated information for θ=135 degrees has lower interpolation accuracy than the other pieces of interpolated information, and the difference from the true value is large. The solid line shows the polarization component information I(θ) (=I(θ;α,Imax,Imin)) obtained from equation (1) for the polarization information (α,Imax,Imin) calculated from the interpolated information by the least squares method. The dashed line shows the polarization component information I'(θ) (=I(θ;α',Imax,Imin')) obtained from equation (1) for the true polarization information (α',Imax,Imin').

[0037] The polarization information (α, Imax, Imin) is calculated so as to minimize the difference between the interpolated information for θ=135 degrees, which has low interpolation accuracy, i.e., which has a large difference from the curve of the polarization component information I'(θ), and the curve of the polarization component information I(θ). As a result, there is a large difference between the curves of the polarization component information I(θ) and I'(θ). In other words, the calculation accuracy of the polarization information is reduced due to the interpolated information having low interpolation accuracy.

[0038] In this embodiment, the polarization information is calculated by weighting interpolation information estimated to have a high interpolation accuracy more heavily than interpolation information estimated to have a low interpolation accuracy. For example, to calculate the polarization information by the least squares method using weights from the first to Nth interpolation information, the polarization information (α, Imax, Imin) that minimizes the sum S of the following equation (5) is found.

[0039]

number

[0040] where w j is the j-th interpolation information D j The j-th interpolation information D j and polarization component information I(φj ) contributes relatively little to the sum S of the differences between the polarization information and the polarization information. Therefore, it is possible to prevent the accuracy of calculating the polarization information from being reduced due to interpolation information with low interpolation accuracy.

[0041] FIG. 7 shows an example of how weighted polarization information is calculated. Similar to FIG. 6, FIG. 7 shows four pieces of interpolated information for θ=0°, 45°, 90°, and 135°. The solid line shows the polarization component information I(θ) (=I(θ;α,Imax,Imin)) obtained from Equation (1) for the polarization information (α,Imax,Imin) weighted from the interpolated information and calculated by the least squares method. The dashed line shows the polarization information I'(θ) (=I(θ;α',Imax,Imin')) obtained from Equation (1) for the true polarization information (α',Imax,Imin'). The curve for the polarization component information I(θ) deviates from the polarization component information for θ=135° compared to FIG. 6, but it approaches the curve for the polarization component information I'(θ), improving the accuracy of the polarization information calculation.

[0042] The interpolation accuracy for determining the weights used for interpolation may be estimated based on the distance between the pixel (source pixel) from which the polarization component information used for interpolation is obtained and the pixel to be interpolated (interpolated pixel), for example. Alternatively, the accuracy may be estimated by comparing the polarization component information near the source pixel with the polarization component information near the interpolated pixel. The interpolation accuracy may be estimated each time polarization component information is obtained, or may be estimated in advance.

[0043] The polarization information calculated by the polarization information calculation unit 14 is recorded in the recording unit 15. The polarization information may be recorded independently, but is preferably recorded in association with the interpolation information used in the calculation. [Example]

[0044] FIG. 8 is a diagram showing the polarization pixel array in the polarization imaging element 12 of this embodiment. The polarization pixel array of this embodiment has four pixel groups (θ1 pixel group, θ2 pixel group, θ3 pixel group, and θ4 pixel group). In FIG. 8, the 2×2 pixel area surrounded by a thick line indicates a square array pattern, which is the basic array pattern of this embodiment. In the polarization pixel array of this embodiment, the square array pattern is repeatedly arranged in both the x-axis direction and the y-axis direction. In FIG. 8, the x and y coordinates are assigned so that the coordinates of the bottom left pixel are (0,0) and the coordinates of the top right pixel are (5,5). Polarization component information acquired at pixel (x,y) is represented as θi(x,y), and the j-th interpolated information obtained by interpolating pixel (x,y) is represented as D j (x,y;φ j ) is expressed as

[0045] In this embodiment, the interpolated information for each pixel is calculated as the average of the polarization component information acquired at the nearest pixel. The flow of calculating polarization information in this embodiment will be explained using the calculation of polarization information for the (2,2) pixel as an example. Table 1 shows the nearest pixel of the (2,2) pixel from which each polarization component information can be acquired and the distance between the interpolated pixel. The distance between the interpolated pixels is the distance between the pixel (source pixel) from which the polarization component information used for interpolation is acquired and the pixel to be interpolated (interpolated pixel); for example, the distance between adjacent pixels in the x-axis direction is set to 1.

[0046] [Table 1]

[0047] The interpolation information is calculated by the following equations (6a)-(6d).

[0048]

number

[0049] It is generally considered that the calculation accuracy of the interpolation information (interpolation accuracy) decreases as the distance between the interpolated pixels increases. Therefore, in this embodiment, the inverse of the distance between the interpolated pixels is used as the weight w jThe polarization information is calculated by attaching the above formula to each piece of interpolation information. In other words, the accuracy of the interpolation information based on a source pixel that is a first distance away from the interpolation pixel is estimated to be higher than the accuracy of the interpolated polarization component information based on a source pixel that is a second distance away from the interpolation pixel, which is longer than the first distance. In this case, the weight of the second interpolation information D2 becomes infinite. This is equivalent to calculating the polarization information from the remaining interpolation information under the condition that the polarization component information I(θ2;α,Imax,Imin) calculated from equation (1) using the calculated polarization information (α,Imax,Imin) is equal to the second interpolation information D2(2,2;θ2).

[0050] In this embodiment, the polarization information is calculated using the weighted least squares method, i.e., the polarization information (α, Imax, Imin) that satisfies the following equation (7) and minimizes the sum S of the following equation (8) is found.

[0051]

number

[0052] This example illustrates the calculation of polarization information when θ1 = 0°, θ2 = 45°, θ3 = 90°, and θ4 = 135°, and when interpolation errors occur in the first, third, and fourth interpolated information, resulting in differences from the true values. Table 2 shows the polarization information obtained by the method of this embodiment and the polarization component information calculated from the obtained polarization information. Also shown as a comparative example is the polarization information and polarization component information calculated when no weighting is applied (equal weighting is applied to each interpolated information).

[0053] [Table 2]

[0054] The polarization component information I(θ2) calculated by the method of the comparative example differs from the true value, but the polarization component information I(θ2) calculated by the method of this embodiment does not differ from the true value. Furthermore, the polarization information calculated by the method of this embodiment differs less from the true value than the comparative example, and is calculated with higher accuracy. By similarly weighting pixels other than the (2,2) pixel, polarization information can be calculated with high accuracy. [Example]

[0055] FIG. 9 is a diagram showing the polarization color pixel array in the polarization imaging element 12 of this embodiment. The letters and numbers written on each pixel indicate the type of color filter and polarization filter arranged at that pixel, respectively. Pixels without numbers indicate non-polarized pixels. The polarization color pixel array of this embodiment has five pixel groups (θ1 pixel group, θ2 pixel group, θ3 pixel group, θ4 pixel group, and non-polarized pixel group). The R pixel group and B pixel group are all non-polarized pixel groups. The G pixel group has four pixel groups (θ1 pixel group, θ2 pixel group, θ3 pixel group, and θ4 pixel group). In the following description, an R pixel group and non-polarized pixel group will be referred to as the R non-polarized pixel group, and a G pixel group and θ1 pixel group will be referred to as the G θ1 pixel group, etc. Furthermore, information acquired by the R non-polarized pixel group will be referred to as R non-polarized component information, and information acquired by the G θ1 pixel group will be referred to as G θ1 information, etc.

[0056] In Fig. 9, the 8x8 pixel area surrounded by a thick line indicates a square array pattern, which is the basic array pattern of this embodiment. In the polarization pixel array of this embodiment, the square array pattern is repeatedly arranged in both the x-axis direction and the y-axis direction. In Fig. 8, the x and y coordinates are assigned so that the coordinates of the bottom left pixel are (0,0) and the coordinates of the top right pixel are (7,7). The polarization component information of the polarization direction θi of color C acquired at the (x,y) pixel is represented as Ci(x,y), and the j-th interpolated information obtained by interpolating the (x,y) pixel is represented as D j (x,y;φ j ,C j ) and C j represents the color information of the j-th interpolation information.

[0057] In this embodiment, the interpolated information for each pixel is calculated as the polarization component information acquired from the nearest pixel. The flow of polarization information calculation in this embodiment will be explained using the calculation of G polarization information for the (3,3) pixel as an example. Table 3 shows the nearest pixel of the (3,3) pixel from which each polarization component information can be acquired and the distance between the interpolated pixel.

[0058] [Table 3]

[0059] The interpolation information is calculated by the following equations (9a)-(9e): Since there are two nearest pixels for the θ3 information, two pieces of interpolation information are calculated.

[0060]

number

[0061] In this embodiment, the polarization information is calculated by assigning a weight wj, which is the inverse of the distance between interpolated pixels, to each piece of interpolated information. The polarization information is calculated using the weighted least squares method. That is, the polarization information (α, Imax, Imin) that satisfies the following equation (10) and minimizes the sum S of the following equation (11) is calculated.

[0062]

number

[0063] This example illustrates the calculation of polarization information when θ1 = 0°, θ2 = 45°, θ3 = 90°, and θ4 = 135°, and when interpolation errors occur in the first to fourth interpolated information and there is a difference from the true value. Table 4 shows the polarization information obtained by the method of this example and the polarization component information calculated from the obtained polarization information. As a comparative example, the table also shows the polarization information and polarization component information calculated when no weighting is applied (equal weighting is applied to each interpolated information).

[0064] [Table 4]

[0065] The polarization information calculated by the method of this embodiment has a smaller difference from the true value than the comparative example, and is calculated with higher accuracy. By similarly weighting G pixels other than the (2,2) pixel, it is possible to calculate polarization information with high accuracy. Furthermore, for R and B pixels, the distance between the interpolated pixels of the nearest pixels from which G polarization component information can be obtained is equal, and the interpolation accuracy is considered to be equal for each G polarization component information, so polarization information is calculated without weighting.

[0066] Because only one piece of polarization component information is acquired for each of the R information and B information, it is not possible to calculate the polarization information from only the polarization component information for each color information, but it is possible to calculate it by using the polarization information for G. For example, because it is thought that the polarization direction α changes little depending on the color, the polarization direction α for the R information and B information is set to be equal to the polarization direction α for the G information, and the polarization component information Imax and Imin are determined so that the ratio between the unpolarized component information and the polarization component information Imax and Imin matches for each color. [Example]

[0067] The polarization color pixel array in the polarization imaging element 12 of this embodiment is the same as that of the polarization color pixel array of the second embodiment. In the second embodiment, the calculation of G polarization information for R and B pixels is performed without weighting, but in this embodiment, the interpolation accuracy is estimated based on the R information and B information, and the polarization information is calculated with weighting. Interpolated information interpolated from a direction where the degree of coincidence of predetermined information (pixel values) is high is considered to have higher interpolation accuracy than interpolated information interpolated from a direction where the degree of coincidence is low. In other words, the accuracy of interpolated information based on source pixels corresponding to pixel values ​​that differ from the pixel value of the interpolated pixel by a first difference is estimated to be higher than the accuracy of interpolated information based on source pixels corresponding to pixel values ​​that differ by a second difference greater than the first difference. Therefore, in this embodiment, the smaller the difference in non-polarized component information between the R and B non-polarized pixels in each interpolation direction (=higher the degree of coincidence), the higher the interpolation accuracy of each G polarization component information is estimated to be. Here, the interpolation direction is the direction of the source pixel as seen from the interpolated pixel, and is expressed by a normalized vector. Furthermore, the pixel at coordinates (x, y) expressed by the following equation (12) is calculated in the interpolation direction (v x ,v y ) is the pixel in

[0068]

number

[0069] Note that (x0, y0) are the coordinates of the interpolated pixel, and k is an arbitrary natural number.

[0070] The calculation of G polarization information at the (3,2) pixel will be explained as an example. Fig. 10 is a diagram showing B non-polarized component information at the B pixel. Table 5 shows the nearest pixel to the (3,2) pixel from which each G polarization component information can be obtained, the interpolation direction, the B non-polarized pixel in each interpolation direction, and the difference in B non-polarized component information.

[0071] [Table 5]

[0072] The interpolation information is calculated by the following equations (13a)-(13d).

[0073]

number

[0074] In this embodiment, the inverse of the difference between the non-polarized component information of B is used as the weight w j The polarization information is calculated by adding the above to each piece of interpolated information. The polarization information is calculated using the weighted least squares method. That is, the polarization information (α, Imax, Imin) that satisfies the following equations (14a) and (14b) and minimizes the sum S of the following equation (15) is found.

[0075]

number

[0076] This example illustrates the calculation of polarization information when θ1 = 0°, θ2 = 45°, θ3 = 90°, and θ4 = 135°, and an interpolation error occurs in the second and third interpolated information, resulting in a difference from the true value. The polarization information calculated by the method of this example and the polarization component information calculated from the obtained polarization information are shown in Table 6. Also shown as a comparative example is the polarization information and polarization component information calculated when no weighting is applied (equal weighting is applied to each interpolated information).

[0077] [Table 6]

[0078] The polarization information calculated by the method of this embodiment has a smaller difference from the true value than the comparative example, and is calculated with higher accuracy. [Example]

[0079] FIG. 11 is a diagram showing the polarization color pixel array in the polarization imaging element 12 of this embodiment. Pixels marked with W are pixels of the W pixel group, and are sensitive to the entire R, G, and B wavelength bands. The polarization color pixel array of this embodiment has four pixel groups (θ1 pixel group, θ2 pixel group, θ3 pixel group, and θ4 pixel group). The R pixel group, B pixel group, and G pixel group each have four pixel groups (θ1 pixel group, θ2 pixel group, θ3 pixel group, and θ4 pixel group). The W pixel group has three pixel groups (θ1 pixel group, θ2 pixel group, and θ3 pixel group).

[0080] In Fig. 11, the 12 x 12 pixel area surrounded by a thick line indicates a square array pattern, which is the basic array pattern of this embodiment. In the polarization pixel array of this embodiment, the square array pattern is repeatedly arranged in both the x-axis direction and the y-axis direction. In Fig. 11, the x and y coordinates are assigned so that the coordinates of the bottom left pixel are (0,0) and the coordinates of the top right pixel are (11,11).

[0081] In this embodiment, W information, which has a high pixel ratio, is interpolated and polarization information is calculated, and RGB information is interpolated and polarization information is calculated using the W information. By using W information, which has a high pixel ratio, the accuracy of RGB information interpolation and polarization information calculation is improved.

[0082] The W information is interpolated by averaging the W polarization component information acquired at the nearest pixel. Because three pieces of W interpolated information are calculated for each pixel (W1 information, W2 information, and W3 information), the polarization information is calculated without weighting.

[0083] The interpolated information for RGB information is calculated by the weighted average of the polarization component information I(x, y, θ; W) of W obtained from Equation (1) using the polarization information of W at each pixel and the information acquired from neighboring pixels using the following Equation (16). The neighboring pixels are pixels included in the 9x9 pixel neighborhood centered on the interpolated pixel and are used to acquire the information to be interpolated.

[0084]

number

[0085] Here, l is the number assigned to the neighboring pixel, and the sum is taken for all neighboring pixels. Also, the weight w l is given by the following equation (17) as the product of the weight due to the distance between the interpolated pixels and the weight due to the degree of coincidence Δ of the polarization component information I(x, y, θ; W) of W in a 3×3 pixel neighborhood area centered on the interpolated pixel and the source pixel.

[0086]

number

[0087] Note that σ1 and σ2 are constants. FIG. 12 shows a 9×9 pixel neighborhood (dashed line) centered on the (4,4) pixel and a 3×3 pixel neighborhood (chain line). Also shown are 3×3 pixel neighborhood regions (dotted lines) centered on the (4,3) pixel and the (8,7) pixel, which are neighboring pixels of the (4,4) pixel when interpolating the R1 information. Note that the neighborhood region is not limited to a square as in this embodiment, and may have other shapes. Furthermore, the interpolated pixel or the source pixel of interpolation need not be at the center. In this embodiment, the degree of match Δ is evaluated by the following equation (18).

[0088]

number

[0089] Weight w l The smaller the distance between interpolated pixels and the smaller the matching degree Δ (higher the matching degree), the larger the weight w l It is considered that the interpolated information calculated from the polarization component information having a large value has high interpolation accuracy. Therefore, in this embodiment, the polarization information is calculated by using the weights w used to calculate the interpolated information. l The weighted least squares method is used to calculate the polarization information, with the average of the weights for each interpolated information. This allows for calculation of the polarization information with higher accuracy than when no weighting is used. [Other Examples] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0090] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0091] 13 Interpolation processing unit (acquisition unit) 14 Polarization information calculation unit (calculation unit)

Claims

1. an acquisition unit that acquires a plurality of pieces of interpolated polarization component information for the first pixel based on first to fourth polarization component information acquired from first to fourth pixels, each pixel having a primary sensitivity to a different polarization component; a calculation unit that calculates polarization information at the first pixel by a least squares method using the plurality of pieces of interpolated polarization component information; the calculation unit assigns a greater weight to information estimated to be highly accurate among the plurality of pieces of interpolated polarization component information than to information estimated to be less accurate, and calculates the polarization information including the polarization orientation at which the light intensity of the incident light is maximized, the maximum light intensity of the incident light, and the minimum light intensity of the incident light.

2. The processing device according to claim 1 , wherein the calculation unit estimates the accuracy of each of the plurality of pieces of interpolated polarization component information.

3. 3. The processing device according to claim 2, wherein the calculation unit estimates the accuracy of the interpolated polarization component information based on the polarization component information acquired from a pixel that is a first distance away from the first pixel to be higher than the accuracy of the interpolated polarization component information based on the polarization component information acquired from a pixel that is a second distance away from the first pixel that is longer than the first distance.

4. 3. The processing device according to claim 2, wherein the calculation unit estimates accuracy of the interpolated polarization component information based on polarization component information acquired from pixels corresponding to pixel values ​​that differ from the pixel value of the first pixel by a first difference to be higher than accuracy of the interpolated polarization component information based on polarization component information acquired from pixels corresponding to pixel values ​​that differ from the first pixel by a second difference that is larger than the first difference.

5. 3. The processing device according to claim 2, wherein the calculation unit estimates accuracy of the interpolated polarization component information based on the polarization component information acquired from pixels having a first degree of coincidence between the polarization component information acquired from pixels in a neighboring region and the polarization component information acquired from pixels in a neighboring region of the first pixel to be higher than accuracy of the interpolated polarization component information based on the polarization component information acquired from pixels having a second degree of coincidence between the polarization component information acquired from pixels in a neighboring region of the first pixel and the polarization component information acquired from pixels in a neighboring region of the first pixel that is lower than the first degree of coincidence.

6. 6. The processing device according to claim 5, wherein the pixels in the neighborhood include pixels adjacent to a central pixel.

7. 7. The processing device according to claim 5, wherein the pixels in the neighboring region include pixels that are not adjacent to the central pixel.

8. the acquisition unit acquires a plurality of pieces of interpolated polarization component information for the first pixel based on fifth and sixth polarization component information corresponding to first and second color pixels having primary sensitivities in different wavelength bands, respectively; the sixth polarization component information is obtained using information from the first color pixel; The processing device according to claim 1 , wherein the calculation unit estimates accuracy of information based on the sixth polarization component information among the plurality of pieces of interpolated polarization component information.

9. 9. The processing device according to claim 8, wherein the calculation unit estimates that the accuracy is higher when the degree of coincidence between the polarization component information acquired from pixels in a region adjacent to the first color pixel and the polarization component information acquired from pixels in a region adjacent to the second color pixel is higher.

10. The processing device according to claim 1 , wherein the calculation unit calculates the polarization information using a weighted least squares method.

11. A processing device according to any one of claims 1 to 10; and an imaging element having first to fourth pixels.

12. the imaging element includes first and second color pixel groups; a ratio of pixels included in the first color pixel group to all pixels included in the imaging element is greater than a ratio of pixels included in the second color pixel group to all pixels, the first color pixel group includes three of the first to fourth pixels, 12. The imaging device according to claim 11, wherein the second color pixel group includes the first to fourth pixels.

13. 13. The imaging device according to claim 12, wherein the ratio of pixels included in the first color pixel group to all pixels is 0.5 or more.

14. 14. The imaging device according to claim 12, wherein the wavelength components to which the first color pixel group has primary sensitivity include a green wavelength component.

15. A processing device according to any one of claims 1 to 10; and an optical system.

16. a first step of acquiring a plurality of pieces of interpolated polarization component information at a first pixel based on first to fourth polarization component information acquired from first to fourth pixels each having a primary sensitivity to a different polarization component; a second step of calculating polarization information at the first pixel by a least squares method using the plurality of pieces of interpolated polarization component information; a processing method characterized in that in the second step, a greater weight is assigned to information estimated to be highly accurate among the plurality of pieces of interpolated polarization component information than to information estimated to be less accurate, and the polarization information including the polarization orientation at which the light intensity of the incident light is maximum, the maximum light intensity of the incident light, and the minimum light intensity of the incident light is calculated.

17. A program causing a computer to execute the processing method according to claim 16.

Images