Information generating device, information generating method and program

Abstract

The information generating device (30) obtains observation values ​​for at least three (m≥3) polarization directions from the polarization imaging unit (20). The noise amount calculating unit (35‑1) calculates the noise amount based on the observation value of the first polarization direction. Similarly, the noise amount calculating units (35‑2 to 35‑m) calculate the noise amount based on the observation values ​​of the second polarization direction to the m-th polarization direction. The polarization model estimating unit (36) estimates the polarization model using the observation value for each polarization direction and the noise amount calculated by the noise calculating units (35‑1 to 35‑m). Therefore, a polarization model that is robust to noise can be calculated.

Description

Technical Field

[0001] The technology relates to an information generating device, an information generating method, and a program, and calculates a polarization model that is robust against noise. Background Art

[0002] When separating a reflection component or the like by using polarization, it has been practiced to fit the intensity of the transmitted light transmitted through the polarizer to a cosine curve as a polarization model. For example, in Patent Document 1, separation of a non-polarized component and a polarized component as a reflection component is performed based on the fitted cosine curve. In addition, in Patent Document 2, the polarization characteristics of an object are obtained with high sensitivity by calculating a low-sensitivity polarization model based on a low-sensitivity polarization image and calculating a high-sensitivity polarization model having the same phase component as the low-sensitivity polarization model from an unsaturated image in a high-sensitivity polarization image.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent No. 4974543

[0006] Patent Document 2: WO 2017 / 085993 A Summary of the invention

[0007] Problems to be solved by the present invention

[0008] By the way, it is known that in the case of calculating a high-sensitivity polarization model from an unsaturated image in a high-sensitivity polarization image, in the function fitting by the least square method, when the variation of the observed value follows a normal distribution with the same variance, the fitting accuracy is good. Therefore, for example, when the polarization image includes shot noise of an image sensor in which the variance varies depending on the brightness, the accuracy of the calculated polarization model decreases.

[0009] Therefore, an object of the present technology is to provide an information generating device, an information generating method, and a program capable of calculating a polarization model that is robust against noise.

[0010] Solution to the problem

[0011] The first aspect of the technology is an information generating device, the information generating device comprising:

[0012] a noise amount calculation unit that calculates the noise amount based on the observation value for each of a plurality of polarization directions; and

[0013] A polarization model estimation unit estimates a polarization model by using the observation value and the noise amount calculated by the noise amount calculation unit.

[0014] In this technology, by using a noise variance model showing the relationship between the observation value and the variance for each of a plurality of polarization directions, the variance depending on the observation value is calculated as the noise amount. The noise variance model may be stored in advance, or by detecting a plurality of observation areas such as a flat area or a stationary area according to a captured image, and obtaining the variance according to a plurality of observation values ​​and an average observation value in each of the observation areas, the noise variance model may be generated according to the average observation value and the variance in the plurality of observation areas. The information generating device estimates the polarization model by using the observation value and the calculated noise amount, or detects the observation value in a saturated state, and estimates the polarization model without using the observation value in the saturated state.

[0015] In addition, a horizontal correction is performed to make the observation values ​​of each of the multiple polarization directions obtained for each of the multiple driving modes with different sensitivities or exposure times consistent, and the noise amount is calculated by using the corrected observation value, the correction gain used in the horizontal correction, and the noise variance model. In addition, the polarization model can be estimated by using the corrected observation value and the calculated noise amount.

[0016] For each of the observation values ​​in the plurality of polarization directions, the observation values ​​in which the polarization directions are three or more directions are used. In addition, a polarization imaging unit may be included, which acquires the observation values ​​for each of the plurality of polarization directions. In addition, the polarization imaging unit may store a noise variance model, and the noise amount calculation unit may calculate the noise amount by using the noise variance model stored in the polarization imaging unit.

[0017] The second aspect of this technology is:

[0018] A method for generating information, comprising:

[0019] calculating the noise amount by a noise amount calculation unit based on the observation value for each of the plurality of polarization directions; and

[0020] The polarization model is estimated by the polarization model estimation unit by using the observation value and the noise amount calculated by the noise amount calculation unit.

[0021] The third aspect of this technology is:

[0022] A program causing a computer to perform generation of information based on an observation value for each of a plurality of polarization directions,

[0023] The program causes the computer to:

[0024] a process of calculating the amount of noise based on the observed value for each of a plurality of polarization directions; and

[0025] The process of estimating a polarization model by using observations and a calculated amount of noise.

[0026] Note that the program of the present technology is, for example, a program that can be provided to a general-purpose computer capable of executing various program codes through a storage medium, a communication medium provided in a computer-readable form, for example, a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable form, processing is implemented according to the program on the computer.

[0027] Effects of the Invention

[0028] According to this technology, the amount of noise is calculated based on the observed values ​​for each of a plurality of polarization directions and the polarization model is estimated using the observed values ​​and the calculated amount of noise. Therefore, it becomes possible to calculate a polarization model that is robust to noise. Note that the advantageous effects described in this specification are only examples, and the advantageous effects of the present technology are not limited to these effects and may include additional effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a diagram illustrating the configuration of an information generating system.

[0030] Figure 2 is a diagram illustrating the configuration of a polarization imaging unit.

[0031] Figure 3 It is a diagram illustrating the configuration of the first embodiment of the information generating unit.

[0032] Figure 4 is a flowchart showing the operation of the first embodiment.

[0033] Figure 5 is a diagram illustrating observation values ​​for each region that should have the same brightness.

[0034] Figure 6 It is a diagram illustrating the configuration of the second embodiment of the information generating unit.

[0035] Figure 7 is a flowchart showing the operation of the second embodiment.

[0036] Figure 8 It is a diagram illustrating the configuration of a third embodiment of the information generating unit.

[0037] Fig. 9 is a diagram illustrating the operation of the noise variance model estimation unit.

[0038] Fig.10 is a flowchart showing the operation of the third embodiment.

[0039] Fig.11 : is a diagram illustrating the operation of the noise variance model estimation unit in the fourth embodiment.

[0040] Fig.12 is a flowchart showing the operation of the fourth embodiment.

[0041] Fig.13 It is a diagram illustrating the configuration of a fifth embodiment of an information generating unit.

[0042] Fig.14 is a diagram illustrating the operation of the horizontal correction unit.

[0043] Fig.15 is a flowchart showing the operation of the fifth embodiment.

[0044] Fig.16 It is a diagram illustrating the configuration of a sixth embodiment of an information generating unit.

[0045] Fig.17 is a flowchart showing the operation of the sixth embodiment. DETAILED DESCRIPTION

[0046] The following is a description of an embodiment for implementing the present technology. Note that the description will be made in the following order.

[0047] 1. Information generation system

[0048] 2. Polarization imaging unit

[0049] 2-1. Noise in polarization images

[0050] 3. Information generation unit

[0051] 3-1. First embodiment

[0052] 3-2. Coefficients of the noise variance model

[0053] 3-3. Second Embodiment

[0054] 3-4. Third Embodiment

[0055] 3-5. Fourth embodiment

[0056] 3-6. Fifth embodiment

[0057] 3-7. Sixth Embodiment

[0058] 3-8. Other Implementation Methods

[0059] 4. Information obtained from the polarization model

[0060] 5. Application Examples

[0061] <1. Information Generation System>

[0062] When observing a scene while rotating the polarization element, the observed value (brightness) changes according to the direction of the polarization element. Here, when the rotation angle (polarization angle) of the polarization element is "υ", the change in the observed value can be represented by the polarization model shown in equation (1).

[0063] I(υ)=Asin(2υ)+Bcos(2υ)+C···(1)

[0064] Figure 1 The configuration of the information generating system is illustrated. The information generating system 10 includes a polarization imaging unit 20 and an information generating unit 30. The polarization imaging unit 20 acquires a polarization image of a desired object, and the information generating unit 30 estimates a polarization model for the desired object based on the polarization image acquired by the polarization imaging unit 20, and calculates parameters A, B, and C of equation (1).

[0065] <2. Polarization Imaging Unit>

[0066] The polarization imaging unit 20 acquires an observation value for each of a plurality of polarization directions. Figure 2 The configuration of the polarization imaging unit is illustrated. Figure 2 As shown in (a), the polarization imaging unit 20 includes a polarization filter 202 having a pixel configuration with multiple polarization directions arranged in an image sensor 201, and the polarization imaging unit 20 acquires a polarization image. Note that, Figure 2 (a) illustrates a case where a polarization filter 202 is arranged on the front surface of an image sensor 201, wherein each pixel is any one of four pixels of different polarization directions (polarization directions are indicated by arrows). If the polarization imaging unit is configured as described above, observation values ​​for each of a plurality of polarization directions can be acquired in units of 2×2 pixel blocks. In addition, if interpolation processing is performed by using pixel values ​​along the same polarization direction and observation values ​​at pixel positions along different polarization directions are calculated, observation values ​​for each of a plurality of polarization directions can be acquired for each pixel. In addition, for example, Figure 2As shown in (b), the polarization imaging unit 20 can obtain a polarization image for each of a plurality of polarization directions by setting a rotatable polarization plate 205 as indicated by an arrow on the front surface of the image sensor 201 and performing imaging for each of a plurality of rotational positions of the polarization plate 205, with the lens optical axis of the polarization imaging unit 20 as the rotation axis. If the polarization imaging unit is configured as described above, for an object that does not move, an observation value for each of a plurality of polarization directions of a pixel can be obtained. In addition, although not shown, by using a configuration in which polarization plates with different polarization directions are provided in front of a plurality of imaging units, a plurality of polarization images with different polarization directions can be generated by sequentially performing imaging using a plurality of imaging units from the same position.

[0067] <2-1. Noise of polarization image>

[0068] The polarization imaging unit 20 is configured by using an image sensor as described above, and the noise generated in the image sensor is classified into fixed pattern noise and random noise. Fixed pattern noise is noise generated at a fixed position on the image regardless of the time of imaging. Fixed pattern noise is generated due to changes in the characteristics of the amplification circuit of the photodiode. Random noise is mainly generated by optical shot noise and thermal noise. Optical shot noise is generated by the fact that the number of photons reaching the photodiode of the image sensor fluctuates statistically. Although the optical shot noise is modeled by Poisson distribution, the optical shot noise is represented as a normal distribution in an area where the brightness is large to some extent. Thermal noise is noise generated by the fact that free electrons in a resistor in an amplifier circuit vibrate irregularly due to thermal motion, and is represented by a normal distribution.

[0069] Therefore, the observed random noise of the image sensor is observed as a distribution obtained mainly by adding together the normal distribution of the optical shot noise and the normal distribution of the thermal noise. That is, the noise generated in the image sensor is represented by the normal distribution shown in equation (2), where the standard deviation σ(i), variance σ 2 (i) and the average value i as parameters. Note that in the case of observing the brightness by using an image sensor, the average value i of the distribution corresponds to the true brightness value.

[0070] [Expression 1]

[0071]

[0072] Furthermore, it is known that the variance due to shot noise increases linearly with respect to the brightness value, and in the present invention, the variance σ is converted to 2(i) is represented as the parameter noise variance model shown in equation (3). Note that the coefficient c0 of the noise variance model corresponds to the variance of the distribution of thermal noise, and the multiplication result of the coefficient c1 and the observation value i corresponds to the variance of the distribution of shot noise. In addition, the values ​​of the coefficients c0 and c1 of the noise variance model vary according to the drive parameters (e.g., gain and exposure time) of the image sensor. Note that each imaging operation performed by switching the drive parameters is also called a drive mode.

[0073] σ 2 (i) = c0 + c1·i···(3)

[0074] <3. Information generation unit>

[0075] The information generating unit 30 includes at least a noise amount calculating unit and a polarization model estimating unit. The noise amount calculating unit calculates the variance of the observation values ​​of each of the multiple polarization directions acquired by the polarization imaging unit 20. In the present technology, it is considered that the variance of the observation value is generated by noise, and the variance of the observation value, i.e., the variance of the noise, is calculated as the noise amount. The polarization model estimating unit uses the observation value and the noise amount calculated by the noise amount calculating unit to estimate the polarization model in units of blocks as a group of polarization pixels or in units of pixels, in which the difference with the observation value acquired by the polarization imaging unit is small. Specifically, the polarization model estimating unit calculates the noise amount generated in the image sensor by using a noise variance model based on the polarization image acquired by the polarization imaging unit 20, estimates the polarization model based on the polarization image and the noise amount, and calculates the parameters A, B, and C of equation (1). Since the three parameters A, B, and C are calculated, the information generating unit 30 uses the observation values ​​along at least three or more polarization directions (the polarization directions may include non-polarization). Furthermore, the information generating unit 30 calculates parameters by using the observed values ​​and variances in three or more polarization directions.

[0076] <3-1. First embodiment>

[0077] Figure 3 The configuration of the first embodiment of the information generating unit is illustrated. The information generating unit 30 includes noise amount calculating units 35-1 to 35-m and a polarization model estimating unit 36. The noise amount calculating units 35-1 to 35-m are respectively provided for the first polarization brightness acquiring unit 201-1 to the mth polarization brightness acquiring unit 201-m of the polarization imaging unit, which generates pixel values ​​for calculation of parameters. For example, in the case where the polarization imaging unit 20 has Figure 2In the case of the configuration shown in (a), the parameters are calculated by using the observation values ​​(brightness values) of four pixels having different polarization directions in a two-dimensional area of ​​2×2 pixels, the four pixels having different polarization directions correspond to the first polarization brightness acquisition unit 201-1 to the fourth polarization brightness acquisition unit 201-4. In addition, in the case of the polarization imaging unit 20 having Figure 2 In the case of the configuration shown in (b), and parameters are calculated by using observation values ​​along four directions with different polarization directions for each pixel, wherein the pixels at each of the four rotation positions where the polarization directions of the polarization plates are different from each other are the first polarization brightness acquisition unit 201-1 to the fourth polarization brightness acquisition unit 201-4.

[0078] The noise amount calculation units 35-1 to 35-m each calculate the variance of the noise as the noise amount. The values ​​of the coefficients c0 and c1 of the noise variance model are pre-stored in the noise amount calculation units 35-1 to 35-m, and the noise amount calculation units 35-1 to 35-m each calculate the variance based on equation (3), wherein the observed value for each pixel is taken as the average value i and the variance is output to the polarization model estimation unit 36. Note that the details of the coefficients c0 and c1 of the noise variance model will be described later.

[0079] The polarization model estimation unit 36 ​​calculates the parameters A, B, and C of the polarization model using the observation values ​​(brightness values) obtained by the polarization imaging unit 20 and the noise amounts calculated by the noise amount calculation units 35-1 to 35-m, so that the difference between the polarization model weighted according to the noise amounts and the observation values ​​obtained by the polarization imaging unit is minimized. Specifically, equation (4) is calculated using the weighted least squares method and the parameters A, B, and C are calculated so that the square error is minimized. Equation (5) shows the matrix F used in equation (4). Equation (6) shows the matrix I used in equation (4), and the matrix I shows the observation values ​​obtained by the polarization imaging unit 20. In addition, equation (7) shows the matrix W used in equation (4), the diagonal components in the matrix W are the inverse of the variance, and weighting is performed according to the inverse of the variance. Note that "F T ” is the transposed matrix of the matrix F, and () -1 is the inverse matrix.

[0080] [Expression 2]

[0081]

[0082] Figure 42 is a flowchart showing the operation of the first embodiment. In step ST1, the information generation unit acquires a polarization image. The information generation unit 30 acquires a polarization image showing observation values ​​in a plurality of polarization directions generated by the polarization imaging unit 20 and proceeds to step ST2.

[0083] In step ST2, the information generating unit calculates the noise amount. The information generating unit 30 calculates the noise amount based on equation (3) by using preset coefficients c0 and c1 and the observed value, and proceeds to step ST3.

[0084] In step ST3, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by, for example, the least square method, by using the observation value of the polarization image acquired in step ST1 and the noise amount calculated in step ST2.

[0085] As described above, according to the first embodiment, it becomes possible to estimate a polarization model that is robust against noise generated in the image sensor.

[0086] <3-2. Coefficients of the noise variance model>

[0087] Next, a description will be given of coefficients c0 and c1 used in the noise amount calculation units 35-1 to 35-m. The noise variance model estimation unit estimates a noise variance model by using a polarized captured image of a chart having various brightnesses acquired using an image sensor set in a predetermined drive mode and calculates coefficients c0 and c1 of the noise variance model.

[0088] The noise variance model estimation unit measures the variation of the observed value from, for example, the average observed value of a region in the graph that should have the same brightness and each observed value in the region. Figure 5 The observed values ​​for each area which should have the same brightness are illustrated. The graph MK imaged by the image sensor 201 is Figure 5 , and the graph MK includes, for example, four areas Pa1 to Pa4 having different brightness. Figure 5 In (b), for each of the average values ​​ia1 to ia4 of the regions Pa1 to Pa4, the observed value for each region of the graph is shown. Assuming that the variation of the observed value due to noise follows a normal distribution, there is a characteristic that when the actual observed value is the center, about 99% of the variation falls within three times the standard deviation σ of the normal distribution. Based on this characteristic, it is possible to estimate at what standard deviation the observed value varies. For each of the average values ​​ia1 to ia4 of the regions Pa1 to Pa4, in Figure 5The variance is shown in (c) of . The noise variance model estimation unit performs linear regression to calculate coefficients c0 and c1 of the noise variance model shown in equation (3), where the correspondence between the average values ​​ia1 to ia4 of the regions Pa1 to Pa4 and the variances of the regions Pa1 to Pa4 is fitted into a straight line.

[0089] <3-3. Second Embodiment>

[0090] Figure 6 The configuration of the second embodiment of the information generating unit is illustrated. The information generating unit 30 has a configuration in which saturation detecting units 31-1 to 31-m are further provided for the first embodiment. Similar to the noise amount calculating units 35-1 to 35-m, saturation detecting units 31-1 to 31-m are provided for the first polarization brightness acquiring unit 201-1 to the m-th polarization brightness acquiring unit 201-m of the polarization imaging unit, respectively, which generates pixel values ​​for calculation of parameters.

[0091] The saturation detection units 31-1 to 31-m each detect an observation value in a saturated state. The saturation detection units 31-1 to 31-m each compare the observation value with the maximum output value of the image sensor, and determine the observation value as the maximum output value as a pixel in a saturated state (referred to as a saturated pixel). The saturation detection unit 31-1 outputs saturated pixel information indicating the detected saturated pixel to the noise amount calculation unit 35-1. Similarly, the saturation detection units 31-2 to 31-m output saturated pixel information indicating the detected saturated pixel to the noise amount calculation units 35-2 to 35-m.

[0092] The noise amount calculation units 35-1 to 35-m each calculate the variance based on the saturated pixel information and the observed value for each pixel. The noise amount calculation units 35-1 to 35-m each set the variance of the pixel determined as the saturated pixel in the saturated pixel information to a predetermined value, so that the polarization model estimation unit 36 ​​estimates the polarization model by excluding the observed value in the saturated state. Specifically, the variance of the pixel determined as the saturated pixel is set to infinity, and the inverse of the variance in the matrix W shown in equation (7) is set to "0", thereby estimating the polarization model by excluding the observed value in the saturated state.

[0093] The polarization model estimation unit 36 ​​performs calculation of equation (4) using the observation value (brightness value) acquired by the polarization imaging unit 20 and the noise amount calculated by the noise amount calculation units 35-1 to 35-m, and calculates parameters A, B and C of the polarization model by the least squares method.

[0094] Figure 72 is a flowchart showing the operation of the second embodiment. In step ST11, the information generation unit acquires a polarization image. The information generation unit 30 acquires a polarization image showing observation values ​​in a plurality of polarization directions generated by the polarization imaging unit 20 and proceeds to step ST12.

[0095] In step ST12, the information generating unit 30 detects a saturated pixel. The information generating unit 30 sets a pixel whose observation value is the maximum output value of the image sensor as a saturated pixel, and proceeds to step ST13.

[0096] In step ST13, the information generating unit calculates the noise amount. Based on the saturated pixel detection result in step ST12, the information generating unit 30 calculates the variance of the unsaturated pixels based on equation (2) by using preset coefficients c0 and c1 and the average value i of the observed values, and sets the variance of the saturated pixels to infinity. The information generating unit calculates the variance of the saturated pixels and the variance of the unsaturated pixels as the noise amount, and proceeds to step ST14.

[0097] In step ST14, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by using the observation value of the polarization image acquired in step ST11 and the noise amount calculated in step ST13 by, for example, the least square method.

[0098] As described above, similarly to the first embodiment, according to the second embodiment, it becomes possible to estimate a polarization model robust to noise generated in an image sensor. In addition, according to the second embodiment, saturated pixels are excluded in the estimation of the polarization model, so that the polarization model can be accurately estimated.

[0099] <3-4. Third Embodiment>

[0100] Next, in the third embodiment, the noise variance model is estimated so that the processing of calculating the coefficients c0 and c1 in advance is not required. The noise variance model is estimated based on the relationship between the average observation value and the variance in the multiple observation areas by detecting multiple observation areas from the polarization image acquired by the polarization imaging unit 20 and obtaining the variance from the average observation value of each of the multiple observation values ​​and the observation area.

[0101] Figure 8The configuration of the third embodiment of the information generating unit is illustrated. The information generating unit 30 includes a noise variance model estimating unit 33, noise amount calculating units 35-1 to 35-m, and a polarization model estimating unit 36. The noise amount calculating units 35-1 to 35-m are respectively provided for the first polarization brightness acquiring unit 201-1 to the mth polarization brightness acquiring unit 201-m of the polarization imaging unit, which generates observation values ​​for calculation of parameters.

[0102] The noise variance model estimation unit 33 estimates the noise variance model based on the observation values ​​obtained by the first polarization brightness acquisition unit 201-1 to the mth polarization brightness acquisition unit 201-m. Based on the observation values ​​obtained by the first polarization brightness acquisition unit 201-1 to the mth polarization brightness acquisition unit 201-m, the noise variance model estimation unit 33 detects a plurality of flat areas without texture (for example, areas along a two-dimensional direction where the difference between the observation values ​​is within a preset threshold), and calculates the mean and variance of the observation values ​​for each of the flat areas. It is assumed that the flat area is, for example, an area where the difference between the observation values ​​is within a preset threshold. In addition, the noise variance model estimation unit 33 performs linear regression to calculate the coefficients c0 and c1 of the noise variance model shown in equation (3), wherein the correspondence between the mean and variance of the plurality of flat areas is fitted into a straight line. The noise variance model estimation unit 33 outputs the calculated coefficients c0 and c1 to the noise amount calculation units 35-1 to 35-m. Note that if the noise variance model estimation unit 33 estimates the noise variance model by using observation values ​​obtained using an image sensor set in different driving modes, the change in the average value is greater than the change in the average value in the case of one driving mode, and it becomes possible to accurately estimate the noise variance model.

[0103] Fig. 9 The operation of the noise variance model estimation unit in the third embodiment is illustrated. The noise variance model estimation unit 33 detects flat areas from polarization images along multiple polarization directions. For example, Fig. 9 (a) of FIG. 1 illustrates four flat areas Pb1 to Pb4 detected from the polarization image. For each of the average values ​​ib1 to ib4 of the flat areas Pb1 to Pb4, Fig. 9 The observed values ​​are shown in (b) of FIG. 1 . Note that for the average value of the flat region, if the average value for each of a plurality of drive modes is used, rather than being limited to one drive mode, the distribution of the average value can be expanded. For each of the average values ​​ib1 to ib4 of the flat regions Pb1 to Pb4, Fig. 9 The variance is shown in (c). Fig. 9As shown in (c), the noise variance model estimation unit 33 performs linear regression to calculate the coefficients c0 and c1 of the noise variance model shown in equation (3), where the correspondence between the average values ​​ib1 to ib4 of the flat areas Pb1 to Pb4 and the variances of the flat areas Pb1 to Pb4 is fitted into a straight line.

[0104] The noise amount calculation units 35-1 to 35-m each calculate the variance of the noise as the noise amount. The noise amount calculation units 35-1 to 35-m each calculate the variance based on equation (3) by using the coefficients c0 and c1 calculated by the noise variance model estimation unit 33, where the observation value for each pixel is the average value i in equation (3). The noise amount calculation units 35-1 to 35-m each output the calculated variance as the noise amount to the polarization model estimation unit 36.

[0105] Similar to the first embodiment, the polarization model estimation unit 36 ​​calculates parameters A, B, and C of the polarization model by the least square method using the observation values ​​acquired by the polarization imaging unit 20 and the noise amounts calculated by the noise amount calculation units 35 - 1 to 35 - m.

[0106] Fig.10 3 is a flowchart showing the operation of the third embodiment. In step ST21, the information generation unit acquires a polarization image. The information generation unit 30 acquires a polarization image showing observation values ​​in a plurality of polarization directions generated by the polarization imaging unit 20 and proceeds to step ST22.

[0107] In step ST22, the information generating unit detects a flat area. The information generating unit 30 detects a plurality of flat areas without texture from the polarization image, and proceeds to step ST23.

[0108] In step ST23, the information generating unit estimates the noise variance model. The information generating unit 30 performs linear regression by calculating the mean and variance of the observation values ​​for each of the plurality of flat regions detected in step ST22, and calculates the coefficients c0 and c1 of the noise variance model shown in equation (3), and proceeds to step ST24.

[0109] In step ST24, the information generating unit calculates the noise amount. The information generating unit 30 calculates the variance as the noise amount based on equation (3) by using the coefficients c0 and c1 calculated in step ST22 and the observation value, and proceeds to step ST25.

[0110] In step ST25, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by using the observation value of the polarization image acquired in step ST21 and the noise amount calculated in step ST23 by, for example, the least square method.

[0111] As described above, similarly to the first embodiment, according to the third embodiment, even if the parameters of the noise variance model are not stored in advance, the noise variance model can be estimated, and it becomes possible to estimate a polarization model robust to noise generated in the image sensor.

[0112] <3-5. Fourth embodiment>

[0113] Next, the fourth embodiment will be described. In the third embodiment, multiple flat areas are detected as multiple observation areas, and the noise variance model is estimated by using the observation values ​​of the flat areas, but in the fourth embodiment, multiple stationary areas are detected as observation areas based on multiple polarization images in the time direction.

[0114] The fourth embodiment of the information generating unit is configured similarly to the third embodiment, and the operation of the noise variance model estimating unit 33 is different from that of the third embodiment.

[0115] The noise variance model estimation unit 33 estimates the noise variance model based on, for example, the observation values ​​of the multiple frames acquired by the first polarization brightness acquisition unit 201-1 to the mth polarization brightness acquisition unit 201-m and the driving mode of the image sensor. For example, based on the observation values ​​of the multiple frames acquired by the first polarization brightness acquisition unit 201-1 to the mth polarization brightness acquisition unit 201-m, the noise variance model estimation unit 33 detects multiple non-moving static areas (for example, areas where the difference in the time direction of the observation values ​​at the same position is within a preset threshold), and calculates the mean and variance of the observation values ​​for each of the static areas. In addition, the noise variance model estimation unit 33 performs a linear regression in which the correspondence between the mean and variance of the multiple static areas is fitted into a straight line, and calculates the coefficients c0 and c1 of the noise variance model shown in equation (3). The noise variance model estimation unit 33 outputs the calculated coefficients c0 and c1 to the noise amount calculation units 35-1 to 35-m.

[0116] Fig.11 The operation of the noise variance model estimation unit in the fourth embodiment is illustrated. Fig.11 (a) of FIG. 3 shows a polarization image group of multiple frames. The noise variance model estimation unit 33 detects a stationary area from the polarization image group. Assume Fig.11(a) of FIG. 3 illustrates polarization images at time t1 and time t2, and the noise variance model estimation unit 33 detects, for example, four stationary regions Pc1 to Pc4. For each of the average values ​​ic1 to ic4 of the stationary regions Pc1 to Pc4, Fig.11 The observed values ​​are shown in (b) of FIG. 1 . For each of the average values ​​ic1 to ic4 of the stationary regions Pc1 to Pc4, Fig.11 The variance is shown in (c). Fig.11 As shown in (c), the noise variance model generation unit 33 performs linear regression in which the correspondence between the average values ​​ic1 to ic4 of the stationary areas Pc1 to Pc4 and the variances of the stationary areas Pc1 to Pc4 are fitted into a straight line, and calculates the coefficients c0 and c1 of the noise variance model shown in equation (3).

[0117] The noise amount calculation units 35-1 to 35-m each calculate the variance of the noise as the noise amount. The noise amount calculation units 35-1 to 35-m each calculate the variance based on equation (3) by using the coefficients c0 and c1 calculated by the noise variance model estimation unit 33, where the observation value for each pixel is the average value i in equation (3). The noise amount calculation units 35-1 to 35-m each output the calculated variance as the noise amount to the polarization model estimation unit 36.

[0118] Similar to the first embodiment, the polarization model estimation unit 36 ​​calculates parameters A, B, and C of the polarization model by the least square method using the observation values ​​acquired by the polarization imaging unit 20 and the noise amounts calculated by the noise amount calculation units 35 - 1 to 35 - m.

[0119] Fig.12 : is a flowchart showing the operation of the fourth embodiment. In step ST31, the information generating unit acquires polarization images of a plurality of frames. The information generating unit 30 acquires polarization images of a plurality of frames generated by the polarization imaging unit 20, and proceeds to step ST32.

[0120] In step ST32, the information generating unit detects a still area. The information generating unit 30 detects a plurality of still areas from the polarization images of the plurality of frames acquired in step ST31, and proceeds to step ST33.

[0121] In step ST33, the information generating unit estimates the noise variance model. The information generating unit 30 performs linear regression by calculating the mean and variance of the observation values ​​for each of the plurality of stationary regions detected in step ST32, calculates coefficients c0, c1 of the noise variance model shown in equation (3), and proceeds to step ST34.

[0122] In step ST34, the information generating unit calculates the noise amount. The information generating unit 30 calculates a variance as the noise amount based on equation (3) by using the coefficients c0 and c1 calculated in step ST33 and the observation value, and proceeds to step ST35.

[0123] In step ST35, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by using the observation value of the polarization image acquired in step ST31 and the noise amount calculated in step ST33 by, for example, the least square method.

[0124] As described above, similarly to the third embodiment, according to the fourth embodiment, even if the parameters of the noise variance model are not stored in advance, the noise variance model can be estimated, and it becomes possible to estimate a polarization model robust to noise generated in the image sensor.

[0125] <3-6. Fifth embodiment>

[0126] Next, in the fifth embodiment, the polarization model is estimated with a higher dynamic range than the first to fourth embodiments. The polarization imaging unit 20 of the fifth embodiment acquires polarization images in multiple driving modes by switching the driving mode and performing imaging in multiple driving modes. The information generating unit 30 estimates the polarization model based on the polarization images in multiple modes acquired by the polarization imaging unit 20 based on the observed values ​​and the sensitivity to the observed values.

[0127] The polarization imaging unit 20 is configured to be able to change the sensitivity or exposure time, and acquire a polarization image whenever the sensitivity or exposure time changes. Figure 2 In the case of the configuration shown in (a), imaging can be performed in a plurality of different drive modes to obtain a plurality of polarization images, or the exposure time for each pixel can be made variable to obtain a polarization image including a plurality of observation values ​​at different exposure times in one imaging. In addition, by setting a neutral density filter having a different light attenuation amount for each pixel, a polarization image including a plurality of observation values ​​at different sensitivities can be obtained through one imaging. In addition, in the case where the polarization imaging unit 20 has Figure 2 In the case of the configuration shown in (b), for example, an operation of acquiring a polarization image by setting the polarization direction of the polarization plate in a desired plurality of directions is performed in a plurality of different drive modes. Note that the polarization imaging unit 20 only needs to acquire a plurality of polarization images with different sensitivities or exposure times, and is not limited to the above configuration.

[0128] Fig.13 The configuration of the fifth embodiment of the information generation unit is illustrated. Note that Fig.13 The following case is shown: a pixel group for obtaining observation values ​​along m types of polarization directions is provided as the polarization imaging unit 20 for each of n types of driving modes, and observation values ​​are obtained along the first polarization direction υ1 to the m-th polarization direction υm for each of the first driving mode to the n-th driving mode. In addition, the observation value along the first polarization direction in the first driving mode is “i 1,1 ”, the observed value along the second polarization direction in the first driving mode is “i 1,2 ”……The observed value along the mth polarization direction in the nth driving mode is “i n,m ”.

[0129] The information generating unit 30 includes horizontal correction units 32 - 1 , 1 to 32 - 1 , m , . . . , 32 - n , 1 to 32 - n , m , noise amount calculating units 35 - 1 , 1 to 35 - 1 , m , . . . , 35 - n , 1 to 35 - n , m , and a polarization model estimating unit 36 ​​.

[0130] The horizontal correction units 32-1,1 to 32-1,m, ..., 32-n,1 to 32-n,m each perform horizontal correction to make the observation values ​​for each of the multiple polarization directions consistent, and the observation values ​​are obtained for each of the multiple operation modes. For example, the horizontal correction unit 32-1,1 uses a gain to correct the observation value that depends on the sensitivity and exposure time of the acquired polarization image, and outputs the corrected observation value to the noise amount calculation unit 35-1,1 and the polarization model estimation unit 36. Similarly, the horizontal correction units 32-2,1 to 32-2,m, ..., 32-n,1 to 32-n,m use a gain to correct the observation value that depends on the sensitivity and exposure time of the acquired polarization image, and output the corrected observation value to the noise amount calculation unit 35-2,1 to 35-2,m, ..., 35-n,1 to 35-n,m and the polarization model estimation unit 36. The horizontal correction units 32-1,1 to 32-1,m, . . . , 32-n,1 to 32-n,m set gains so that observation values ​​for each polarization direction are equal to each other regardless of sensitivity or exposure time.

[0131] Fig.14 The operation of the horizontal correction unit is illustrated, and it is noted that Fig.14 1 shows the operation of the horizontal correction units 32-1,1 to 32-1,m and 32-n,1 to 32-n,m, for example, in the case where the number of polarization directions is m. For example, the horizontal correction unit 32-1,1 converts the observation value “i 1 ” along the polarization direction v1 obtained by the polarization brightness acquisition unit of the first sensitivity in the polarization imaging unit 20 into 1,1” is multiplied by r1. Similarly, the horizontal correction units 32-1,2 to 32-1,m convert the observation values ​​“i along the polarization directions υ2 to υm” acquired by the polarization brightness acquisition unit of the first sensitivity into 1,2 、i 1,3 、……、i 1,m " is multiplied by r1. In addition, the horizontal correction unit 32-n,1 converts the observation value "i" along the polarization direction v1 obtained by the polarization brightness acquisition unit of the nth sensitivity in the polarization imaging unit 20 n,1 Multiply by r n Similarly, the horizontal correction units 32-n,2 to 32-n,m convert the observation values ​​"i" along the polarization directions υ2 to υm acquired by the polarization brightness acquisition unit of the nth sensitivity into n,2 、i n,3 、……、i n,m Multiply by r n Therefore, the observed value after level correction is “r1×i 1,1 =r n ×i n,1 ”, “r1×i 1,2 =r n ×i n,2 ”, “r1×i 1,3 =r n ×i n,3 ”, …, “r1×i 1,m =r n ×i n,m ”.

[0132] The noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m each calculate the variance of the noise as the noise amount. In the case where the observation value i is corrected to the observation value "r×i", the noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m set the variance to "r 2 ×σ 2 (i)”. The noise amount calculation units 35-1,1 to 35-1,m, . . . , 35-n,1 to 35-n,m output the calculated variances as noise amounts to the polarization model estimation unit 36.

[0133] The polarization model estimation unit 36 ​​calculates the parameters A, B and C of the polarization model based on equation (8) using the observed values ​​after horizontal correction of the observed values ​​acquired by the polarization imaging unit 20 and the noise amounts calculated by the noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m. Equation (9) shows the matrix F used in equation (8). Equation (10) shows the matrix I used in equation (8), and the matrix I shows the observed values ​​acquired by the polarization imaging unit 20. In addition, equation (11) shows the matrix W used in equation (8), and the matrix W is a diagonal matrix showing the inverse of the variance.

[0134] [Expression 3]

[0135]

[0136] Fig.15 : is a flowchart showing the operation of the fifth embodiment. In step ST41, the information generation unit acquires polarization images in a plurality of driving modes. The information generation unit 30 acquires polarization images in a plurality of driving modes generated by the polarization imaging unit 20, and proceeds to step ST42.

[0137] In step ST42, the information generating unit performs horizontal correction. The information generating unit 30 performs horizontal correction on the polarization image acquired in step ST41 so that the observation values ​​of the polarization images in the plurality of driving modes are equal to each other at the same pixel position, and proceeds to step ST43.

[0138] In step ST43, the information generating unit calculates the noise amount. The information generating unit 30 calculates the variance of the noise as the noise amount based on equation (3) by using preset coefficients c0 and c1 and the observed value corrected in step ST42, and proceeds to step ST44.

[0139] In step ST44, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by using the observed value corrected in step ST42 and the noise amount calculated in step ST43 by, for example, the least square method.

[0140] As described above, according to the fifth embodiment, a polarization model having a high dynamic range and in which noise is maximally suppressed can be estimated from polarization images acquired in a plurality of different driving modes.

[0141] <3-7. Sixth embodiment>

[0142] Next, in the sixth embodiment, the polarization model is estimated with a higher dynamic range than the first to fourth embodiments without using saturated pixels. The polarization imaging unit 20 of the sixth embodiment acquires polarization images in multiple driving modes by switching the driving mode and performing imaging in multiple driving modes. The information generating unit 30 estimates the polarization model without using saturated pixels based on the polarization images for each of the multiple driving modes acquired by the polarization imaging unit 20.

[0143] The polarization imaging unit 20 is configured to be able to change the sensitivity or the exposure time similarly to the fifth embodiment, acquires a polarization image every time the sensitivity or the exposure time is changed, and outputs the acquired polarization image to the information generating unit 30 .

[0144] Fig.16 The configuration of the sixth embodiment of the information generating unit is illustrated. Note that the case where the polarization imaging unit 20 acquires observation values ​​along the first polarization direction υ1 to the m-th polarization direction υm in each of the first driving mode to the n-th driving mode is shown. In addition, the observation value along the first polarization direction in the first driving mode is “i 1,1 ”, the observed value along the second polarization direction in the first driving mode is “i 1,2 ”……The observed value along the mth polarization direction in the nth driving mode is “i n,m ”.

[0145] The information generating unit 30 includes a saturation detecting unit 31-1,1 to 31-1,m, ..., 31-n,1 to 31-n,m, a horizontal correcting unit 32-1,1 to 32-1,m, ..., 32-n,1 to 32-n,m, a noise amount calculating unit 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m and a polarization model estimating unit 36.

[0146] The saturation detection unit 31-1,1 determines whether the observation value along the first polarization direction in the first driving mode is in a saturated state, and outputs the determination result to the noise amount calculation unit 35-1,1. The saturation detection unit 31-1,2 determines whether the observation value along the second polarization direction in the first driving mode is in a saturated state, and outputs the determination result to the noise amount calculation unit 35-1,2. The saturation detection units 31-1,3 to 31-1,m, ..., 31-n,1 to 31-n,m also similarly perform determination of saturated pixels, and output the determination results to the noise amount calculation units 35-1,3 to 35-1,m, ..., 35-n,1 to 35-n,m.

[0147] The horizontal correction unit 32-1,1 corrects the observation value depending on the sensitivity and exposure time of the acquired polarization image using the gain, and outputs the corrected observation value to the noise amount calculation unit 35-1,1 and the polarization model estimation unit 36. Similarly, the horizontal correction units 32-1,2 to 32-1,m, ..., 32-n,1 to 32-n,m also similarly correct the observation value depending on the sensitivity and exposure time of the acquired polarization image using the gain, and output the corrected observation value to the noise amount calculation unit 35-1,2 to 35-1,m, ..., 35-n,1 to 35-n,m and the polarization model estimation unit 36. The horizontal correction units 32-1,1 to 32-1,m, ..., 32-n,1 to 32-n,m set the gain so that the observation values ​​for each polarization direction are equal to each other regardless of the sensitivity or exposure time.

[0148] The noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m each calculate the variance of the noise as the noise amount. When the observation value i is corrected to the observation value r×i, the noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m set the variance to “r 2 ×σ 2 (i)". In addition, in the case where the pixel is determined to be a saturated pixel by the saturation detection unit 31-1,1, the noise amount calculation unit 35-1,1 sets the variance to infinity. Similarly, in the case where the pixel is determined to be a saturated pixel by the saturation detection units 31-1,2 to 31-1,m, ..., 31-n,1 to 31-n,m, the noise amount calculation units 35-1,2 to 35-1,m, ..., 35-n,1 to 35-n,m set the variance to infinity. The noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m output the calculated variance as the noise amount to the polarization model estimation unit 36.

[0149] The polarization model estimation unit 36 ​​calculates parameters A, B and C of the polarization model using the observation values ​​after horizontal correction of the observation values ​​obtained by the polarization imaging unit 20 and the noise amounts calculated by the noise amount calculation units 35-1,1 to 35-1,m, ..., 35-n,1 to 35-n,m.

[0150] Fig.17 : is a flowchart showing the operation of the sixth embodiment. In step ST51, the information generating unit acquires polarization images in a plurality of driving modes. The information generating unit 30 acquires polarization images in a plurality of driving modes generated by the polarization imaging unit 20, and proceeds to step ST52.

[0151] In step ST52, the information generating unit 30 detects a saturated pixel. The information generating unit 30 sets a pixel whose observation value is the maximum output value of the image sensor as a saturated pixel, and proceeds to step ST53.

[0152] In step ST53, the information generating unit performs horizontal correction. The information generating unit 30 performs horizontal correction on the polarization image acquired in step ST51 so that the observation values ​​of the polarization images in the plurality of driving modes are equal to each other at the same pixel position, and proceeds to step ST54.

[0153] In step ST54, the information generating unit calculates the noise amount. The information generating unit 30 calculates the variance as the noise amount based on equation (3) by using preset coefficients c0 and c1 and the observed value corrected in step ST53. In addition, the information generating unit 30 sets the variance of the saturated pixel detected in step ST52 to infinity, and proceeds to step ST55.

[0154] In step ST55, the information generating unit calculates parameters. The information generating unit 30 calculates parameters of the polarization model by using the observed value corrected in step ST53 and the noise amount calculated in step ST54 by, for example, the least square method.

[0155] As described above, similar to the fifth embodiment, according to the sixth embodiment, a polarization model having a high dynamic range and in which noise is maximally suppressed can be estimated based on polarization images acquired under a plurality of different driving modes. In addition, according to the sixth embodiment, saturated pixels are excluded in the estimation of the polarization model, so that the polarization model can be accurately estimated. Note that in the case where the gain of the horizontal correction unit is high and the corrected observation value is in a saturated state, pixels in a saturated state can be excluded as saturated pixels in the estimation of the polarization model.

[0156] <3-8. Other embodiments>

[0157] The polarization imaging unit 20 and the information processing unit 30 are not limited to being separately provided, and the polarization imaging unit 20 and the information processing unit 30 may be integrally configured, and one of the polarization imaging unit 20 and the information processing unit 30 may be included in the other.

[0158] In addition, in the above-mentioned embodiment, the following case is described: the case where the coefficients c0 and c1 of the noise variance model are pre-stored in the noise amount calculation unit, or the case where the noise variance model estimation unit calculates the coefficients c0 and c1 of the noise variance model. However, the information generation unit 30 can obtain the coefficients c0 and c1 of the noise variance model from the polarization imaging unit 20. For example, in the case where the polarization imaging unit 20 and the information processing unit 30 are separately provided, the noise variance model is estimated when the polarization imaging unit 20 is manufactured, and the coefficients c0 and c1 are stored. In addition, when the observation value is output, the polarization imaging unit 20 outputs the coefficients c0 and c1 of the noise variance model as additional information. When the normal model is estimated based on the observation value, the information generation unit 30 calculates the noise amount by using the coefficients c0 and c1 of the noise variance model indicated as the additional information of the observation value. If such a configuration is used, it becomes easy to estimate the normal model that is robust to noise even when a different polarization imaging unit 20 is switched and used.

[0159] In addition, in the fifth embodiment, a horizontal correction unit and a noise amount calculation unit are set for each driving mode and each polarization direction, and in the sixth embodiment, a saturation detection unit, a horizontal correction unit, and a noise amount calculation unit are set for each driving mode and each polarization direction; however, for example, in a case where a polarization image is acquired by performing multiple imaging while changing the driving mode, a horizontal correction unit, a noise amount calculation unit, and a saturation detection unit for each polarization direction may be set only for one driving mode to sequentially perform processing for each driving mode, and the polarization model estimation unit may estimate a normal model by using the observed value and the noise amount of each driving mode. If such a configuration is used, the information generation unit can be simply configured.

[0160] <4. Information obtained from the polarization model>

[0161] Next, the information obtained from the polarization model will be illustrated. When the coordinate system changes, the polarization model equation shown in equation (1) becomes equation (12). The polarization degree p in equation (12) is calculated based on equation (13), and the azimuth angle is calculated based on equation (14). Note that the degree of polarization ρ indicates the amplitude of the polarization model equation, and the azimuth Indicates the phase of the polarization model equation.

[0162] [Expression 4]

[0163] I(υ)=C·(1+ρ·cos(2(υ-φ))…(12)

[0164]

[0165] It is known that if the polarization degree and refractive index of an object are clear, the zenith angle θ can be calculated (e.g., WO2016 / 088483A). Therefore, the indicated azimuth angle can be generated by using the polarization model equation and normal information of the zenith angle θ. In addition, the parameter C indicates the average of the observed values, and the degree of polarization ρ indicates the amplitude of the polarization model equation. Therefore, the diffuse reflection component can be calculated by subtracting the maximum amplitude from the average value. In addition, the specular reflection component can be calculated by subtracting the diffuse reflection component from the observed value. That is, by estimating a polarization model that is robust to noise using the technology according to the contents of the present disclosure, normal information that is robust to noise can be generated and the reflection component can be calculated, so that, for example, a reflection-reduced image with a higher S / N ratio than before can be obtained, and the surface direction of the object can be estimated with higher accuracy than before. In addition, an arbitrary polarization direction image (pseudo PL filter image) with a higher S / N ratio than before can be obtained.

[0166] <5. Application Examples>

[0167] The technology according to the content of the present disclosure can be applied to various fields. The present technology can be applied to devices installed on any type of mobile body, for example, automobiles, electric vehicles, hybrid vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, robots, etc. In addition, the present technology can be applied to devices installed on devices used in the production process of a factory or on devices used in the construction field. When the present technology is applied to such a field, by generating normal information based on an estimated polarization model, for example, the surrounding environment can be accurately grasped in three dimensions, and the fatigue of the driver or worker can be reduced. In addition, it becomes possible to perform autonomous driving, etc. more safely.

[0168] The technology according to the present disclosure can also be applied to the medical field. For example, if normal information is generated, the specular reflection component is removed based on the estimated polarization model, etc., the three-dimensional shape of the surgical site or a reflection-free image can be accurately obtained, and it becomes possible to reduce the fatigue of the surgeon and perform the operation safely and more reliably. In addition, the technology according to the present disclosure can be applied to fields such as public services. For example, when an image of an object is published in a book, magazine, etc., unnecessary reflection components, etc. can be accurately removed from the image of the object based on the estimated polarization model.

[0169] The series of processing steps described in the specification may be performed by hardware, software, or a combination of both. In the case of performing processing by software, a program recording the processing sequence is installed in a memory in a computer incorporated in dedicated hardware, and the program is executed. Alternatively, the program may be installed and executed in a general-purpose computer capable of performing various types of processing.

[0170] For example, the program may be pre-recorded in a read-only memory (ROM), a solid-state drive (SSD), or a hard disk as a recording medium. Alternatively, the program may be temporarily or permanently stored (recorded) in a removable recording medium such as a floppy disk, a compact disk read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disk (DVD), a Blu-ray Disc (Blu-ray Disc, a registered trademark) (BD), a magnetic disk, or a semiconductor memory card. Such a removable recording medium may be provided as so-called packaged software.

[0171] In addition, in addition to being installed to a computer from a removable recording medium, the program can also be transmitted to the computer wirelessly or by wire from a download site via a network such as a local area network (LAN) or the Internet. The computer can receive the program sent in this manner and install the program in a recording medium such as a built-in hard disk.

[0172] Note that the advantageous effects described in this specification are only examples, and the advantageous effects of the present technology are not limited to these examples, but may include additional effects not described herein. In addition, the present technology should not be interpreted as being limited to the embodiments of the above-mentioned technology. The embodiments of the technology disclose the present technology by example, and it should be obvious that those skilled in the art can modify those embodiments or replace them with other embodiments without departing from the scope of the present technology. In other words, the claims should be considered when understanding the subject matter of the present technology.

[0173] Furthermore, the information generating device of the present technology may also have the following configuration.

[0174] (1) An information generating device comprising:

[0175] a noise amount calculation unit that calculates the noise amount based on the observation value for each of a plurality of polarization directions; and

[0176] A polarization model estimation unit estimates a polarization model by using the observation value and the noise amount calculated by the noise amount calculation unit.

[0177] (2) The information generating device according to (1), wherein the noise amount calculation unit calculates the noise amount by using a noise variance model showing a relationship between the observed value and variance.

[0178] (3) The information generating device according to (2), wherein the noise amount calculation unit sets a variance corresponding to the observation value as the noise amount.

[0179] (4) The information generating device according to any one of (2) to (3), wherein the noise variance model is pre-stored in the noise amount calculation unit.

[0180] (5) The information generating device according to any one of (2) to (3), further comprising:

[0181] A noise variance model estimation unit detects a plurality of observation areas according to a captured image, obtains a variance according to a plurality of observation values ​​and an average observation value in each of the observation areas, and estimates a noise variance model according to the average observation value and the variance in the plurality of observation areas, wherein

[0182] The noise amount calculation unit calculates the noise amount by using the noise variance model estimated by the noise variance model estimation unit.

[0183] (6) The information generating device according to (5), wherein the noise variance model estimation unit sets a flat area in the captured image as the observation area.

[0184] (7) The information generating device according to (5), wherein the noise variance model estimation unit sets a stationary region in the captured image as the observation region.

[0185] (8) The information generating device according to any one of (1) to (7), further comprising:

[0186] A saturation detection unit detects observations in a saturated state, where

[0187] The polarization model estimation unit excludes the observation value in the saturation state and then estimates the polarization model.

[0188] (9) The information generating device according to (8), wherein:

[0189] The noise amount calculation unit sets the noise amount of the observation value in the saturation state in the saturation detection unit to a predetermined value, and

[0190] The polarization model estimation unit excludes the observation value in the saturation state by setting the noise amount to the predetermined value, and then estimates the polarization model.

[0191] (10) The information generating device according to any one of (1) to (9), further comprising:

[0192] a horizontal correction unit that performs horizontal correction to make the observation values ​​for each of the plurality of polarization directions consistent, the observation values ​​being acquired for each of the plurality of driving modes, wherein

[0193] the noise amount calculation unit that calculates the noise amount by using the observation value corrected by the level correction unit, a correction gain used in the level correction, and the noise variance model, and

[0194] The polarization model estimation unit estimates the polarization model by using the observation value corrected by the horizontal correction unit and the noise amount calculated by the noise amount calculation unit.

[0195] (11) The information generating device according to (10), wherein the plurality of driving modes are driving modes having different sensitivities or exposure times.

[0196] (12) The information generating device according to any one of (1) to (11), wherein the observation value for each of the plurality of polarization directions is an observation value in which the polarization directions are three or more directions.

[0197] (13) The information generating device according to any one of (1) to (12), further including: a polarization imaging unit that acquires an observation value for each of the plurality of polarization directions.

[0198] (14) The information generating device according to (13), wherein:

[0199] The polarization imaging unit stores a noise variance model, and

[0200] The noise amount calculation unit calculates the noise amount by using the noise variance model stored in the polarization imaging unit.

[0201] Industrial Applicability

[0202] In the information generating device, the information generating method and the program of the technology, the noise amount is calculated based on the observation values ​​for each of the plurality of polarization directions, and the polarization model is estimated by using the observation values ​​and the calculated noise amount. Therefore, it becomes possible to calculate a polarization model that is robust to noise. Therefore, it is suitable for a device that uses normal information, reflection components, etc. calculated based on the polarization model.

[0203] Reference numerals list

[0204] 10 Information Generation System

[0205] 20 Polarization Imaging Unit

[0206] 30 Information generation unit

[0207] 31-1 to 31-m, 31-1,1 to 31-n,m saturation detection unit

[0208] 32-1,1 to 32-n,m horizontal correction units

[0209] 33 Noise variance model estimation unit

[0210] 35-1 to 35-m, 35-1,1 to 35-n,m Noise amount calculation unit

[0211] 36 Polarization model estimation unit

[0212] 201 Image Sensor

[0213] 202 Polarization filter

[0214] 205 Polarizing Plate

Claims

1. An information generating device, comprising: a noise amount calculation unit that calculates a variance of noise as a noise amount based on an observed value of brightness for each of a plurality of polarization directions; as well as a polarization model estimating unit that estimates a polarization model by using the observed value and the noise amount calculated by the noise amount calculating unit; in, the noise amount calculation unit calculates the noise amount by using a noise variance model showing a relationship between the observed value of the brightness and the variance of the noise; The noise amount calculation unit sets a variance of an observation value corresponding to the brightness as the noise amount; and The polarization model estimation unit is configured to calculate parameters of the polarization model by using the observed value of the brightness and the noise amount calculated by the noise amount calculation unit so as to minimize a difference between a polarization model weighted according to the noise amount and the observed value of the brightness.

2. The information generating device according to claim 1, wherein: The noise variance model is pre-stored in the noise amount calculation unit.

3. The information generating device according to claim 1, further comprising: A noise variance model estimation unit detects a plurality of observation areas from a captured image, obtains a variance from a plurality of observation values ​​and an average observation value in each of the observation areas, and generates a noise variance model from the average observation value and the variance in the plurality of observation areas, wherein The noise amount calculation unit calculates the noise amount by using the noise variance model estimated by the noise variance model estimation unit.

4. The information generating device according to claim 3, wherein: The noise variance model estimation unit sets a flat area in the captured image as the observation area.

5. The information generating device according to claim 3, wherein: The noise variance model estimation unit sets a stationary area in the captured image as the observation area.

6. The information generating device according to claim 1, further comprising: A saturation detection unit detects observations in a saturated state, where The polarization model estimation unit excludes the observation value in the saturation state and then estimates the polarization model.

7. The information generating device according to claim 6, wherein: The noise amount calculation unit sets the noise amount of the observation value in the saturation state in the saturation detection unit to a predetermined value, and The polarization model estimation unit excludes the observation value in the saturation state by setting the noise amount to the predetermined value, and then estimates the polarization model.

8. The information generating device according to claim 1, further comprising: a horizontal correction unit that performs horizontal correction to make the observation values ​​for each of the plurality of polarization directions consistent, the observation values ​​being acquired for each of the plurality of driving modes, wherein the noise amount calculation unit calculates the noise amount by using the observation value corrected by the level correction unit, a correction gain used in the level correction, and the noise variance model, and The polarization model estimation unit estimates the polarization model by using the observation value corrected by the horizontal correction unit and the noise amount calculated by the noise amount calculation unit.

9. The information generating device according to claim 8, wherein: The plurality of driving modes are driving modes having different sensitivities or exposure times.

10. The information generating device according to claim 1, wherein: The observation value for each of the plurality of polarization directions is an observation value in which the polarization directions are three or more directions.

11. The information generating device according to claim 1, further comprising: A polarization imaging unit acquires an observation value for each of the plurality of polarization directions.

12. The information generating device according to claim 11, wherein: The polarization imaging unit stores a noise variance model, and The noise amount calculation unit calculates the noise amount by using the noise variance model stored in the polarization imaging unit.

13. A method for generating information, comprising: calculating, by a noise amount calculation unit, a variance of noise as a noise amount based on an observed value of brightness for each of a plurality of polarization directions; as well as estimating, by a polarization model estimation unit, a polarization model by using the observation value and the noise amount calculated by the noise amount calculation unit; in, the noise amount calculation unit calculates the noise amount by using a noise variance model showing a relationship between the observed value of the brightness and the variance of the noise; The noise amount calculation unit sets a variance of an observation value corresponding to the brightness as the noise amount; and The polarization model estimation unit is configured to calculate parameters of the polarization model by using the observed value of the brightness and the noise amount calculated by the noise amount calculation unit so as to minimize a difference between a polarization model weighted according to the noise amount and the observed value of the brightness.

14. A recording medium having stored therein a program causing a computer to execute generation of information based on an observed value of brightness for each of a plurality of polarization directions, The program causes the computer to implement the method according to claim 13.

Images