Information processing device, information processing method, and program

Abstract

An achromatic color region extraction unit 31 uses polarization information acquired from a color polarized image to calculate achromatic color determination information for each color component of the color polarized image and outputs a region for which the achromatic color determination information satisfies conditions for being an achromatic color as an achromatic color region. On the basis of the white balance gain for the achromatic color region extracted by the achromatic color region extraction unit 31 to be an achromatic color, a gain setting unit 32 sets a white balance gain to be used for white balance adjustment of the color polarized image. For example, the gain setting unit 32 sets a white balance gain for the entire color polarized image on the basis of the white balance gain for the achromatic color region or sets a white balance gain for a region that is not an achromatic color region on the basis of the white balance gain for a plurality of achromatic color regions. The present invention makes it possible to easily perform white balance adjustment on the basis of polarization information.

Description

[Technical Field]

[0001] This technology relates to an information processing device, an information processing method, and a program, and enables easy adjustment of white balance. [Background technology]

[0002] Conventionally, when capturing an image of a subject using an imaging device, white balance adjustment is performed so that a white object appears white. In white balance adjustment, a white balance gain (hereinafter simply referred to as "gain") is automatically set for each color component, taking into account the light source of the captured scene, and the signal level of each color component is adjusted using the set gain. For example, a method known as "gray world" performs white balance adjustment by assuming that the average pixel value of the entire image is an achromatic color. Furthermore, as shown in Patent Document 1, if a method is used that estimates the light source color using a reflection model of an object, white balance adjustment can be performed based on the estimated light source color. Furthermore, Non-Patent Document 1 proposes a method of performing white balance adjustment using deep learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4447520 [Non-patent literature]

[0004] [Non-Patent Document 1] Afifi, Mahmoud and Brown, Michael S. Deep White-Balance Editing, CVPR 2020. Summary of the Invention [Problem to be solved by the invention]

[0005] However, the gray world method treats the average color of the image as the light source color, so accuracy drops when the hypothesis does not hold, such as when the average color of the object is biased. Also, because it uses the average of the entire image, it is difficult to handle scenes where the light source color differs from region to region.

[0006] Furthermore, the method for estimating light source color disclosed in Patent Document 1 assumes that the luminance values ​​in an image are composed of "diffuse light only" or "diffuse light + specular light," extracts the specular light, and treats it as the light source color. However, in reality, the intensity of diffuse light varies depending on the texture and object normal, so it becomes necessary to select a pair that is not affected by these factors. Furthermore, the assumption of "diffuse light only" is very strict, and it is difficult to find pixels for which this assumption holds. Furthermore, while methods using deep learning can utilize context such as objects in the image, the environment, and the time of day, they rely on training data and require very heavy and complex processing.

[0007] Therefore, an object of this technology is to provide an information processing device, an information processing method, and a program that enable easy white balance adjustment using a color polarization image. [Means for solving the problem]

[0008] The first aspect of this technology is an achromatic region extractor that extracts achromatic regions from a color polarization image using polarization information acquired from the color polarization image; a gain setting unit that sets a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the achromatic color region extracted by the achromatic color region extraction unit achromatic; An information processing device comprising:

[0009] In this technology, the achromatic region extraction unit extracts achromatic regions from the color polarized image using polarization information acquired from the color polarized image. For example, the achromatic region extraction unit uses the polarization information to calculate the degree of linear polarization for each color component of the color polarized image as achromatic determination information, and identifies regions where the variation in the degree of linear polarization between color components falls within a predetermined achromatic region determination criterion as achromatic regions. The achromatic region extraction unit also calculates a Stokes vector as achromatic determination information, and identifies regions where the variation between color components in the ratio of multiple components of the Stokes vector falls within a predetermined achromatic region determination criterion as achromatic regions. In this case, the achromatic region extraction unit uses at least a component indicating the luminance or average luminance of unpolarized light as the multiple components.

[0010] The gain setting unit sets the white balance gain used in white balance adjustment of the color polarization image to a white balance gain that will cause the achromatic regions extracted by the achromatic region extraction unit to become achromatic. The white balance gain for regions other than the achromatic regions is set by interpolation using the white balance gain set for each achromatic region. The gain setting unit also sets the white balance gain to be used for the entire region of the color polarization image, or the white balance gain for regions other than the achromatic region, based on the white balance gain set for each achromatic region. The gain setting unit may also perform interpolation using the white balance gain of a neighboring achromatic region. For example, the achromatic region extraction unit classifies the extracted achromatic regions, and the gain setting unit sets the white balance gain and position set for each class classified by the classification as the white balance gain and position of a neighboring achromatic region. Furthermore, the gain setting unit may also divide the color polarization image into regions and set the white balance gain for each divided region.

[0011] The gain setting unit may also switch the white balance gain setting for the color polarization image depending on the variation in the white balance gain set for each achromatic region extracted by the achromatic region extraction unit. For example, if the variation in the white balance gain is within a predetermined tolerance range, the gain setting unit sets the white balance gain to be used for all regions of the color polarization image based on the white balance gain set for each achromatic region. If the variation in the white balance gain exceeds the tolerance range, the gain setting unit sets the white balance gain for regions other than the achromatic regions based on the white balance gain set for each achromatic region.

[0012] The second aspect of this technology is extracting an achromatic region from the color polarization image using polarization information acquired from the color polarization image with an achromatic region extractor; a gain setting unit sets a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the achromatic color region extracted by the achromatic color region extraction unit achromatic; The present invention relates to an information processing method including the steps of:

[0013] The third aspect of this technology is A program for causing a computer to perform white balance adjustment, extracting achromatic regions from the color polarization image using polarization information acquired from the color polarization image; a step of setting a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the extracted achromatic color area achromatic; The program is executed by the computer.

[0014] The program of the present technology is, for example, a program that can be provided in a computer-readable format to a general-purpose computer capable of executing various program codes via a storage medium or communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an imaging system. [Figure 2] FIG. 2 is a diagram illustrating the configuration of a polarization imaging unit. [Figure 3] FIG. 10 is a diagram illustrating pixel configurations for a plurality of polarization directions. [Figure 4] FIG. 10 is a diagram illustrating an example of a pixel configuration (in the case where three primary color pixels and a white pixel are provided) with a plurality of polarization directions. [Figure 5] FIG. 10 is a diagram illustrating a pixel configuration for a plurality of polarization directions (when non-polarization pixels are provided). [Figure 6] FIG. 10 is a diagram illustrating an example in which polarization pixel blocks are thinned out. [Figure 7] 10 is a flowchart illustrating an example of the operation of the imaging system. [Figure 8] FIG. 10 is a diagram illustrating an example in which a plurality of achromatic regions are detected. [Figure 9] FIG. 10 is a diagram illustrating an example in which a plurality of achromatic regions are clustered based on their positions. [Figure 10] FIG. 10 is a diagram illustrating an example of an operation when region division is performed. [Figure 11] 10 is a flowchart illustrating a selection operation in a gain setting process. [Figure 12] FIG. 10 is a diagram illustrating an example of gain settings when a plurality of achromatic color regions are extracted. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments of the present technology will be described in the following order. 1. Extraction of achromatic regions using polarization information 2. About the embodiment 2-1. Configuration of the embodiment 2-2. Operation of the embodiment 2-3. Gain setting process

[0017] <1. Extraction of achromatic regions using polarization information> The information processing device of the present technology extracts achromatic regions using polarization information acquired from a color polarized image, and sets the white balance gain (hereinafter simply referred to as "gain") used in white balance adjustment of the color polarized image to a gain that makes the extracted achromatic regions achromatic. Note that the following description will be given for a case in which the color space of the color polarized image is the RGB color space, and the gains are set for each of the three primary color components: R (red), G (green), and B (blue).

[0018] When extracting achromatic regions using polarization information, the information processing device uses a Stokes vector that can represent the polarization state. The components of a Stokes vector are represented by four components s0, s1, s2, and s3. When the polarization state is represented by a Stokes vector, the polarization transformation can be expressed by a Mueller matrix. In the Stokes vector, component s0 represents the luminance or average luminance of unpolarized light. Component s1 represents the difference in intensity (luminance difference) between when the polarization direction of the polarizer is 0° and when it is 90°. Component s2 represents the difference in intensity (luminance difference) between when the polarization direction of the polarizer is 45° and when it is 135°. Component s3 represents the degree of polarization of circularly polarized light. As will be described later, a linear polarizer is used as the polarizer to acquire a color polarization image, and therefore, in this technology, components s0, s1, and s2 are used to extract achromatic regions.

[0019] The Mueller matrix of an object can be expressed as a linear sum of the matrix Ms, which indicates specular reflection, and the matrix Md, which indicates diffuse reflection. In this case, the weights of each color component related to specular reflection are equal and are "ks", and the weights related to diffuse reflection are "kdR" for the red component, "kdG" for the green component, and "kdB" for the blue component.

[0020] In this case, the Stokes vector of the incident light is "kR(s0,s1,s2) T ", kG(s0, s1, s2) T ,kB(s0,s1,s2) T ", the Stokes vector of the observation light can be expressed by equations (1) to (3). Furthermore, if the object is achromatic, the relationship in equation (4) holds.

[0021]

number

[0022] Therefore, the Stokes vector of the observation light is expressed by equations (5) to (7) when the object is achromatic.

[0023]

number

[0024] Furthermore, the degree of linear polarization (DoLP) is expressed by equations (8) to (10). If the object is achromatic, equation (11) holds. Therefore, by searching for an area that satisfies equation (11), an achromatic area can be extracted.

[0025]

number

[0026] Furthermore, the extraction of achromatic regions using polarization information is not limited to using the linear polarization degree DoLP, and other methods may be used. For example, the information processing device extracts achromatic regions using multiple components of a Stokes vector. Component s0 of the Stokes vector indicates the luminance or average luminance of unpolarized light. Components s1 and s2 indicate an intensity difference, and therefore vary more due to the polarization state than component s0. Therefore, one of the multiple components of the Stokes vector, for example, two components, is set to component s0, which indicates the luminance or average luminance of unpolarized light.

[0027] The information processing device calculates the ratio of two components of the Stokes vector for each color component as shown in equations (12) to (14). Furthermore, if the object is achromatic, the relationship in equation (15) holds. Therefore, by searching for an area that satisfies equation (15), an achromatic area can be extracted.

[0028]

number

[0029] <2. About the form of implementation> <2-1. Configuration of the embodiment> 1 illustrates an example of the configuration of an imaging system using an information processing device according to the present technology. The imaging system 10 includes a polarization imaging section 20 and an information processing device 30.

[0030] The polarization imaging unit 20 acquires a polarization image using a polarization element. FIG. 2 illustrates an example of the configuration of the polarization imaging unit. The polarization imaging unit 20 acquires a polarization image by arranging a polarization filter 202 with pixels arranged in multiple polarization directions on an image sensor 201, such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device), as shown in FIG. 2(a). The polarization filter 202 may be, for example, a wire grid or photonic liquid crystal, as long as it can extract linearly polarized light from the subject light. To acquire a color polarization image, a color filter is provided on the incident surface of the image sensor 201. Alternatively, as shown in FIG. 2(b), the polarization imaging unit 20 may generate multiple polarization images with different polarization directions using a multi-lens array configuration. For example, multiple lenses 203 (four in the figure) are provided in front of the image sensor 201, and each lens 203 forms an optical image of the subject on the imaging surface of the image sensor 201. Furthermore, a polarizing plate 204 is provided in front of each lens 203, and the polarization direction of the polarizing plate 204 is set to a different direction, thereby generating multiple polarized images with different polarization directions. By configuring the polarization imaging unit 20 in this manner, multiple polarized images can be acquired with a single imaging session, allowing for rapid recognition processing of the object to be recognized. Furthermore, as shown in Figure 2(c), polarizing plates 212-1 to 212-4 with different polarization directions may be provided in front of the imaging units 210-1 to 210-4, generating multiple polarized images with different polarization directions from multiple different viewpoints.

[0031] In addition, when the subject moves slowly or moves in steps, a polarizing plate 211 may be provided in front of the image capturing unit 210, as shown in (d) of Fig. 2. In this case, the polarizing plate 211 is rotated to capture images in a plurality of different polarization directions, thereby obtaining a plurality of polarized images with different polarization directions.

[0032] In the cases of (b) and (c) of Figure 2, if the positional spacing between each lens 203 and each image capture unit 210-1 to 210-4 is negligibly short relative to the distance to the subject, parallax can be ignored for multiple polarized images with different polarization directions. Therefore, by averaging the brightness of polarized images with different polarization directions, it is possible to obtain an image equivalent to an unpolarized normal brightness image. In addition, if parallax cannot be ignored, it is possible to obtain an image equivalent to an unpolarized normal brightness image by aligning the polarized images with different polarization directions according to the amount of parallax and averaging the brightness of the polarized images after alignment. In addition, in the case of (d) of Figure 2, it is possible to obtain an image equivalent to an unpolarized normal brightness image by averaging the brightness of polarized images with different polarization directions for each pixel.

[0033] 3 to 5 illustrate pixel configurations for multiple polarization directions, with the configurations shown in the figures repeated in the horizontal and vertical directions. (a) and (b) of FIG. 3 illustrate the arrangement of polarization pixels. (a) of FIG. 3 illustrates a 2×2 pixel polarization pixel block configured with polarization pixels having polarization directions (polarization angles) of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, for example. (b) of FIG. 3 illustrates a 4×4 pixel polarization pixel block configured with polarization pixels having polarization directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees, for example, using 2×2 pixels as the polarization direction unit. When the polarization component unit of the polarization filter is 2×2 pixels as shown in (b) of FIG. 3, the proportion of polarization components leaking in from adjacent regions of different polarization component units relative to the polarization components obtained for each polarization component unit is smaller than in the case of the 1×1 pixel shown in (a) of FIG. 3. Furthermore, when a polarizing filter uses a wire grid, polarized light whose electric field component is perpendicular to the direction of the grid (wire direction) is transmitted, and the longer the wire, the higher the transmittance. Therefore, when the polarization component unit is 2x2 pixels, the transmittance is higher than when it is 1x1 pixel. Therefore, when the polarization component unit is 2x2 pixels, the transmittance is higher than when it is 1x1 pixel, and the extinction ratio can be improved.

[0034] Figures 3(c) to 3(g) show examples of pixel configurations for acquiring color polarization images. Figure 3(c) shows a case where the 2x2 pixel polarization pixel block shown in Figure 3(a) is used as one color unit, and the three primary color pixels (red, green, and blue pixels) are arranged in a Bayer array.

[0035] FIG. 3(d) illustrates an example in which three primary color pixels are arranged in a Bayer array for each pixel block of 2×2 pixels with the same polarization direction shown in FIG. 3(b).

[0036] Figure 3(e) illustrates an example in which three primary color pixels are arranged in a Bayer array for each 2x2 pixel block with the same polarization direction, and 2x2 pixel blocks with different polarization directions are pixels of the same color.

[0037] (f) of Figure 3 shows a case where, for a 2x2 pixel block in a Bayer array with the same polarization direction, the phase difference in the polarization direction between horizontally adjacent pixel blocks is 90 degrees and the phase difference in the polarization direction between vertically adjacent pixel blocks is ±45 degrees.

[0038] (g) of Figure 3 shows a case where, for a 2x2 pixel block in a Bayer array with the same polarization direction, the phase difference in the polarization direction between vertically adjacent pixel blocks is 90 degrees and the phase difference in the polarization direction between horizontally adjacent pixel blocks is ±45 degrees.

[0039] Figure 4 illustrates an example in which three primary color pixels and a white pixel are provided. For example, Figure 4(a) illustrates an example in which one green pixel is used as a white pixel in a pixel block of 2x2 pixels in the same polarization direction and arranged in a Bayer array as shown in Figure 3(b).

[0040] (b) of Figure 4 illustrates an example in which, in a pixel block of 2x2 pixels with the same polarization direction and Bayer arrangement shown in (c) of Figure 3, one green pixel is set as a white pixel, and a block of 2x2 pixels with different polarization directions is set as pixels of the same color.

[0041] By providing white pixels in this way, the dynamic range for generating normal information can be expanded compared to when white pixels are not provided, as disclosed in Patent Document "WO 2016 / 136085." Furthermore, because white pixels have a good S / N ratio, calculation of color difference, etc., is less susceptible to noise.

[0042] FIG. 5 illustrates an example in which non-polarizing pixels are provided, and the polarization direction and color pixel display are the same as those in FIG.

[0043] Figure 5(a) shows an example in which a 4x4 pixel block is formed using two 2x2 pixel blocks with four different polarization directions and two 2x2 pixel blocks made of non-polarized pixels, and the pixel blocks of polarized pixels are green pixels, and the pixel blocks of non-polarized pixels are red pixels or blue pixels, with pixel blocks of the same color (2x2 pixels) arranged in a Bayer array.

[0044] Figure 5 (b) illustrates a case where polarized pixels with a 45-degree phase difference are arranged diagonally within a 2x2 pixel block, and the polarization directions of the polarized pixels are two directions with a 45-degree phase difference.This illustrates a case where pixel blocks of the three primary colors are arranged in a Bayer array, with polarized images with two different polarization directions and pixel blocks consisting of two non-polarized pixels as color units.

[0045] Figure 5(c) shows an example in which a 2x2 pixel block is used as a color unit, pixel blocks of the three primary colors are arranged in a Bayer array, and polarized pixels with two different polarization directions are provided in the pixel block of green pixels.

[0046] (d) of Figure 5 illustrates an example in which polarized pixels are arranged in the same manner as (b) of Figure 5, and a pixel block consisting of polarized images with two different polarization directions and two non-polarized pixels has three green pixels and one non-polarized pixel as a red pixel, and an adjacent pixel block has one non-polarized pixel as a blue pixel.

[0047] Figures 5(e) and (f) show cases where non-polarized pixels are used as color pixels and pixels of the three primary colors are arranged in a 4x4 pixel block, while Figures 5(g) and (h) show cases where some of the non-polarized pixels are used as color pixels and pixels of the three primary colors are arranged in a 4x4 pixel block.

[0048] 3 to 5 are merely examples, and other configurations may be used. In addition, in order to enable high-sensitivity imaging even at night, a configuration in which infrared (IR) pixels are mixed and repeated may be used.

[0049] Figure 6 shows an example where the polarization pixel blocks are thinned out. Figure 6(a) shows an example where 4x4 pixel polarization pixel blocks are repeatedly arranged for every 8x8 pixel block. In this case, pixels with the same polarization direction and color are arranged every 8 pixels in both the horizontal and vertical directions.

[0050] Figure 6(b) shows an example where 4x4 pixel polarization pixel blocks are repeatedly arranged for each 16x16 pixel block. In this case, pixels with the same polarization direction and color are arranged at a 16-pixel period in both the horizontal and vertical directions. Note that the polarization pixel blocks may also be arranged so that pixels with the same polarization direction and color are arranged at a 32-pixel period or a 64-pixel period in both the horizontal and vertical directions. Furthermore, the repetition period of pixels with the same polarization direction and color may be different in the horizontal and vertical directions, or may be different between the center and end portions of the image sensor.

[0051] Note that the polarization imaging section 20 that acquires the color polarization image is not limited to the configuration described above, and may have other configurations as long as it can acquire a color polarization image that provides the polarization information used to extract achromatic regions. Furthermore, the color polarization image used by information processing device 30 does not necessarily have to be output from polarization imaging section 20 to information processing device 30. For example, if a color polarization image generated by polarization imaging section 20 or the like is recorded on a recording medium, the color polarization image recorded on the recording medium may be read and output to information processing device 30.

[0052] The information processing device 30 includes an achromatic region extraction unit 31 , a gain setting unit 32 , and a white balance adjustment unit 33 .

[0053] The achromatic region extraction unit 31 of the information processing device 30 uses polarization information of the color polarization image acquired by the polarization imaging unit 20 to extract regions (achromatic regions) representing achromatic objects within the image. The achromatic region extraction unit 31 uses the polarization information to calculate achromatic determination information for each color component of the color polarization image, for example, and extracts regions that satisfy the achromatic determination information as achromatic regions. The achromatic determination information may be calculated by calculating the linear polarization degree DoLP (DoLP) as described above in <1. Regarding Extraction of Achromatic Regions> and extracting regions that satisfy Equation (11) as achromatic regions. Alternatively, the two components of the Stokes vector may be used to divide one component by the other and extract regions where the division result satisfies Equation (15). Note that achromatic regions are not limited to regions that satisfy Equation (11) or Equation (15). Regions where the variation between color components falls within a predetermined achromatic region determination criterion (e.g., within ±α) are also considered achromatic regions. The achromatic region extraction unit 31 outputs the extraction result of the achromatic region to the gain setting unit 32 .

[0054] The gain setting unit 32 sets a gain for white balance adjustment based on the color polarization image acquired by the polarization imaging unit 20 and the achromatic region extraction result output from the achromatic region extraction unit 31. The gain setting unit 32 sets a gain that makes the color polarization image of the achromatic region extracted by the achromatic region extraction unit 31 achromatic. The gain may be set for each color component, or the gain of one color component may be set based on the gain of another color component. The gain setting unit 32 outputs the set gain to the white balance adjustment unit 33.

[0055] The white balance adjustment unit 33 adjusts the gain of the color polarization image acquired by the polarization imaging unit 20 using the gain set by the gain setting unit 32, and outputs the color polarization image with adjusted white balance to an external device, such as a display device or a recording device.

[0056] <2-2. Operation of the embodiment> FIG. 7 is a flowchart illustrating an operation of an imaging system using the information processing device of the present technology.

[0057] In step ST1, the imaging system acquires a color polarization image. Polarization imaging section 20 of imaging system 10 acquires a color polarization image containing polarization information for multiple polarization directions, and then the process proceeds to step ST2.

[0058] In step ST2, the imaging system performs an achromatic region extraction process. The achromatic region extraction unit 31 of the imaging system 10 extracts achromatic regions from the color polarization image based on the polarization information of the color polarization image acquired in step ST1. As explained in <1. Extraction of achromatic regions using polarization information>, the achromatic region extraction unit 31 may extract achromatic regions using the degree of linear polarization (DoLP), or may extract achromatic regions using two components of the Stokes vector. After extracting the achromatic regions from the color polarization image, the process proceeds to step ST3.

[0059] In step ST3, the imaging system performs a gain setting process. Gain setting unit 32 of imaging system 10 calculates a gain for each color component in the color polarization image acquired in step ST1 that will cause the achromatic color region extracted in step ST2 to become achromatic. Based on the calculated gain, gain setting unit 32 sets the gain to be used in white balance adjustment of the color polarization image acquired in step ST1, and then proceeds to step ST4. Details of the gain setting process will be described later.

[0060] In step ST4, the imaging system performs white balance adjustment. White balance adjustment unit 33 of imaging system 10 performs white balance adjustment on the color polarization image acquired in step ST1 by adjusting the levels of each color component using the gain set in step ST3 in the same manner as in the past.

[0061] In this way, according to the present technology, it becomes possible to easily adjust the white balance based on the polarization information acquired from the color polarization image.

[0062] <2-3. Gain setting process> Next, the gain setting process will be described in detail. When achromatic region extraction unit 31 extracts one achromatic region based on polarization information acquired from the color polarization image, gain setting unit 32 calculates a gain for each color component that will make the extracted achromatic region achromatic. Gain setting unit 32 then sets the calculated gain as the gain to be used for all regions in white balance adjustment of the color polarization image.

[0063] Furthermore, when extracting achromatic regions from a color polarization image, the achromatic region extracted by achromatic region extractor 31 is not limited to one region, and multiple regions may be detected. In such cases, gain setting unit 32 may set a gain so that all of the multiple achromatic regions extracted by achromatic region extractor 31 resemble achromatic colors as closely as possible. For example, the gain for each color component may be calculated for each achromatic region, and a statistical value representing the calculated gains, such as the mean, median, or mode, may be calculated for each color component, and this statistical value may be used for all regions of the color polarization image.

[0064] Furthermore, when multiple achromatic regions are detected by the achromatic region extraction unit 31, a gain may be set for each extracted achromatic region. Furthermore, when a gain is set for each extracted achromatic region, an interpolation process may be performed using the gain set for each achromatic region, and a gain may be set for a region different from the achromatic region. Figure 8 illustrates a case where multiple achromatic regions are detected, and Figure 8(a) illustrates a case where four achromatic regions Pw1 to Pw4 are extracted.

[0065] When multiple achromatic regions are detected, the gain setting unit 32 sets the gain for white balance adjustment in regions other than the achromatic regions by interpolation using the gains in the multiple achromatic regions. (b) of Fig. 8 illustrates the gain for the red component, for example. In the achromatic region Pw1, the gain of the red component set to make the achromatic region Pw1 achromatic is "QR1." The gains of the red component set to make the achromatic regions Pw2 to Pw4 achromatic are "QR2" to "QR4."

[0066] The gain setting unit 32 calculates the gain for white balance adjustment in the area Pt different from the achromatic area using a weight according to the distance. Fig. 8(c) shows the distances "Ld1" to "Ld4" from the area Pt different from the achromatic area to the achromatic areas Pw1 to Pw4, respectively.

[0067] In this case, the gain "QRt" of the red component for the region Pt is calculated by the operation of equation (16). Note that the coefficient ka in equation (16) is a coefficient for normalizing the weight as shown in equation (17). The gain setting unit 32 also calculates the gains of the other color components for the region Pt in the same way as for the red component.

[0068]

number

[0069] By performing such processing, the gain setting unit 32 performs interpolation processing using the gain set for each achromatic color region, and sets gains for regions different from the achromatic color region.

[0070] Furthermore, the gain setting unit 32 may set the gain of a region different from the achromatic region based on the gain of a neighboring achromatic region. In this case, the gain setting unit 32 clusters the multiple achromatic regions based on their positions, and performs interpolation using the center of gravity position and representative value of gain for each class as the positions and gains of neighboring achromatic regions, thereby setting the gain of the region different from the achromatic region. The representative value of gain is a value that represents the gain of the achromatic regions within a class, such as the average, median, or mode.

[0071] FIG. 9 illustrates an example where multiple achromatic regions are clustered based on their positions. (a) of FIG. 9 illustrates the clustering results for multiple achromatic regions. For example, suppose the center of gravity of class CL1 is "PW1" and the representative value of the gain of the red component is "QR1." Furthermore, if the center of gravity of class CL2 is "PW2" and the representative value of the gain of the red component is "QR2," the center of gravity of class CL3 is "PW3" and the representative value of the gain of the red component is "QR3," and the center of gravity of class CL4 is "PW4" and the representative value of the gain of the red component is "QR4," then the gains of regions different from the achromatic regions can be set in the same way as described with reference to FIG. 8.

[0072] Alternatively, gain setting unit 32 may divide the color polarization image into regions and set a gain for each divided region. For example, gain setting unit 32 may divide the image into regions using graph cuts or deep learning (such as a convolutional neural network (CNN) or a recurrent neural network (RNN)), and set a single gain for each color component within the divided regions.

[0073] FIG. 10 illustrates an example of the operation when region segmentation is performed. FIG. 10(a) illustrates a color polarization image, and FIG. 10(b) illustrates the region segmentation result. Note that FIG. 10(b) classifies the image into a sky region ARa, a road region ARb, regions ARc1 and ARc2 representing cars, regions ARd1, ARd2, and ARd3 representing the background, and a region ARe representing an achromatic building in the background. Here, if region ARe is extracted as an achromatic region, the gain of the white balance adjustment is set so that region ARe is displayed as an achromatic color. Furthermore, the gain of region ARe is used as the gain of regions ARa, ARb, ARc1, ARc2, ARd1, ARd2, and ARd3.

[0074] The gain setting unit 32 may select one of the above processes. For example, the gain setting unit 32 switches processes depending on the variation in gain of the same color component calculated for each achromatic region, and sets a gain for the entire color polarization image if the variation does not exceed a preset threshold, but sets a gain for each pixel position or for each divided region into which the color polarization image is divided if the variation exceeds the threshold.

[0075] 11 is a flowchart illustrating the selection operation of the gain setting process. In step ST11, the gain setting unit calculates the gain of the extracted achromatic region. The gain setting unit 32 calculates the gain for the pixels of the achromatic region extracted by the achromatic region extraction unit 31, and then the process proceeds to step ST12.

[0076] In step ST12, the gain setting unit 32 determines whether the gain variation is within an allowable range. If the gain variation calculated in step ST11 is within a preset allowable range, that is, if the illumination light is considered to be emitted from a single light source or multiple light sources with a small difference in color temperature, the gain setting unit 32 proceeds to step ST13. If the variation exceeds the allowable range, that is, if the illumination light is considered to be emitted from multiple light sources with different color temperatures, the gain setting unit 32 proceeds to step ST14.

[0077] In step ST13, the gain setting unit 32 sets the gain to be used for all regions of the color polarization image. Gain setting unit 32 sets the gain to be used for all regions of the color polarization image by statistical processing of the gain calculated in step ST11. For example, gain setting unit 32 sets the gain to be used for all regions of the color polarization image to one of the average, mode, or median of the gains calculated in step ST11.

[0078] When the process proceeds from step ST12 to step ST14, the gain setting unit 32 performs a clustering process in step ST14. The gain setting unit 32 performs position-based clustering on the achromatic regions extracted in step ST11, classifies the achromatic regions, and then the process proceeds to step ST15.

[0079] In step ST15, the gain setting unit 32 determines whether the variation in the gain within a class and between classes exceeds a predetermined allowable range. If the variation in the gain within a class and between classes exceeds a predetermined allowable range, the gain setting unit 32 proceeds to step ST16, and if the variation is within the allowable range, the gain setting unit 32 proceeds to step ST17.

[0080] In step ST16, the gain setting unit 32 performs interpolation based on the achromatic region. As described with reference to Fig. 7, the gain setting unit 32 sets the gain of the region different from the achromatic region by interpolation using the gain of the achromatic region calculated in step ST11 and the distance to the achromatic region.

[0081] In step ST17, the gain setting unit 32 performs interpolation processing based on the class. The gain setting unit 32 calculates the center of gravity position and a representative value of the gain for each class classified in step ST14. Furthermore, as described with reference to FIG. 8, the gain setting unit 32 sets the gain for an area different from the achromatic area by interpolation calculation using the representative value of the gain for the class and the distance to the center of gravity position.

[0082] Fig. 12 illustrates an example of gain settings when multiple achromatic regions are extracted. Fig. 12(a) illustrates an example of an imaging scene. For example, a car model OBa and achromatic objects OBc and OBd are placed on a table. Furthermore, illumination light can be irradiated onto the model OBa and objects OBc and OBd on the table from a light (e.g., an incandescent bulb) LT installed in the room. Furthermore, external light (e.g., sunlight) incident through a window LW is irradiated onto the model OBa and objects OBc and OBd.

[0083] Figure 12(b) shows a color polarized image of the imaging area of ​​Figure 12(a), and Figure 12(c) illustrates the region segmentation results. In Figure 12(c), the regions are divided into wall region AR1, table region AR2, floor region AR3, regions AR4 and AR5 representing achromatic objects, and model region AR6. The object in region AR4 is illuminated by illumination light from a light (e.g., an incandescent bulb) LT installed in the room, and the object in region AR5 is illuminated by external light (e.g., sunlight) entering through a window.

[0084] In this case, the color temperatures of the illumination light in regions AR4 and AR5 are different, resulting in variations beyond the allowable range between the gain for region AR4 and the gain for region AR5. Therefore, for example, the gain for region AR6 is set by interpolation based on the gain for region AR4, the gain for region AR5, and the distance to region AR4 and the distance to region AR5. The gain for regions AR1 to AR3 is also set in the same way as for region AR6. This allows for more natural white balance adjustment for regions AR1 to AR3 and AR6, taking into account the two illumination lights. Note that when the illumination light is either light from light source LT or external light, the gains for regions AR4 and AR5 are approximately equal, so the gain is set for the entire color polarization image.

[0085] In this way, according to the present technology, by using polarization information, it is possible to set the white balance gain so that achromatic objects in an image capture scene are displayed in achromatic colors. Furthermore, by using polarization information, it is possible to set the white balance gain so that achromatic objects are displayed in achromatic colors without taking into account the color temperature of the light source, etc. Furthermore, by using polarization information, it is possible to set the white balance gain to an optimal value for each region in an image capture scene. For example, when multiple light sources with different color temperatures are provided, it is possible to set the white balance gain according to the color temperature of the illumination light irradiating the object.

[0086] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer incorporated in dedicated hardware and executed. Alternatively, the program can be installed and executed on a general-purpose computer capable of executing various processes.

[0087] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read-only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called packaged software.

[0088] In addition to being installed on a computer from a removable recording medium, the program may also be transferred wirelessly or by wire from a download site to a computer via a network such as a WAN (Wide Area Network) typified by cellular, a LAN (Local Area Network), or the Internet. The computer can receive the program transferred in this manner and install it on a recording medium such as a built-in hard disk.

[0089] Note that the effects described in this specification are merely examples and are not limiting, and additional effects not described may exist. Furthermore, the present technology should not be interpreted as being limited to the above-described embodiments of the technology. The embodiments of the technology disclose the present technology in the form of examples, and it is obvious that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration when determining the gist of the present technology.

[0090] Furthermore, the information processing device of the present technology can also have the following configuration. (1) an achromatic region extractor that extracts achromatic regions from a color polarization image using polarization information acquired from the color polarization image; a gain setting unit that sets a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the achromatic color region extracted by the achromatic color region extraction unit achromatic; An information processing device comprising: (2) The information processing device described in (1) wherein the achromatic region extraction unit calculates achromatic color determination information for each color component of the color polarization image using the polarization information, and extracts a region where the achromatic color determination information satisfies the conditions for becoming achromatic as the achromatic color region. (3) The information processing device according to (2), wherein the achromatic region extraction unit calculates a degree of linear polarization as the achromatic color determination information. (4) The information processing device according to (3), wherein the achromatic region extraction unit determines, as the achromatic region, a region in which the variation between color components of the linear polarization degree falls within a predetermined achromatic region determination criterion. (5) The information processing device according to (2), wherein the achromatic region extraction unit calculates a Stokes vector as the achromatic color determination information. (6) The information processing device according to (5), wherein the achromatic region extraction unit determines, as the achromatic region, a region in which the variation between color components in the ratio of the multiple components of the Stokes vector is within a predetermined achromatic region determination criterion. (7) The information processing device according to (6), wherein the achromatic region extraction unit uses at least a component indicating unpolarized light luminance or average luminance as the plurality of components. (8) The information processing device according to any one of (1) to (7), wherein the gain setting unit sets a white balance gain to be used in all areas of the color polarization image. (9) The information processing device according to any one of (1) to (7), wherein the gain setting unit sets the white balance gain for each achromatic color region extracted by the achromatic color region extraction unit. (10) The information processing device according to (9), wherein the gain setting unit performs interpolation processing using the white balance gain set for each achromatic color region, and sets a white balance gain for a region different from the achromatic color region. (11) The information processing device according to (10), wherein the gain setting unit performs the interpolation process using a white balance gain of the neighboring achromatic color region. (12) The achromatic region extraction unit classifies the extracted achromatic regions into classes, The information processing device according to (11), wherein the gain setting unit sets the white balance gain and position set for each class classified by the classification to the white balance gain and position of the neighboring achromatic color region. (13) The information processing device according to any one of (9) to (12), wherein the gain setting unit divides the color polarization image into regions and sets the white balance gain for each divided region. (14) An information processing device described in any of (9) to (13), wherein the gain setting unit switches the setting of the white balance gain for the color polarization image in accordance with the variation in the white balance gain set for each achromatic region extracted by the achromatic region extraction unit. (15) The information processing device described in (14), wherein the gain setting unit sets the white balance gain to be used in the entire area of ​​the color polarization image based on the white balance gain set for each achromatic area when the variation in the white balance gain is within a predetermined tolerance range. (16) The information processing device according to (14) or (15), wherein the gain setting unit sets the white balance gain of an area different from the achromatic area based on the white balance gain set for each achromatic area when the variation in the white balance gain exceeds the allowable range. [Explanation of symbols]

[0091] 10. Imaging system 20 Polarization imaging unit 30. Information processing device 31...Achromatic color area extraction part 32 Gain setting section 33 White balance adjustment section 201 Image sensor 202 Polarizing filter 203···Lens 204 Polarizing plate 210, 210-1 to 210-4: Imaging unit 211, 212-1 to 212-4 Polarizing plates

Claims

1. an achromatic region extraction unit that uses polarization information acquired from the color polarization image to calculate a degree of linear polarization or a Stokes vector as achromatic color determination information for each color component of the color polarization image, and extracts achromatic color regions that satisfy the conditions for the achromatic color determination information to be achromatic; a gain setting unit that sets a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the achromatic color region extracted by the achromatic color region extraction unit achromatic; An information processing device comprising:

2. When calculating the degree of linear polarization as the achromatic color determination information, the achromatic color region extraction unit determines, as the achromatic color region, a region in which the variation between color components of the degree of linear polarization falls within a predetermined achromatic color region determination criterion. The information processing device according to claim 1 .

3. When the achromatic region extraction unit calculates a Stokes vector as the achromatic color determination information, the achromatic color region extraction unit determines, as the achromatic color region, a region in which the variation between color components of the ratio of multiple components of the Stokes vector falls within a predetermined achromatic color region determination criterion. The information processing device according to claim 1 .

4. The achromatic region extraction unit uses, as the plurality of components, at least a component indicating unpolarized light luminance or average luminance. The information processing device according to claim 3 .

5. The gain setting unit sets a white balance gain to be used in the entire region of the color polarization image. The information processing device according to claim 1 .

6. The gain setting unit sets the white balance gain for each achromatic color region extracted by the achromatic color region extraction unit. The information processing device according to claim 1 .

7. The gain setting unit performs interpolation processing using the white balance gain set for each of the achromatic color regions, and sets a white balance gain for a region different from the achromatic color region. The information processing device according to claim 6 .

8. The gain setting unit performs the interpolation process using a white balance gain of the neighboring achromatic color region. The information processing device according to claim 7 .

9. The achromatic region extraction unit performs class classification of the extracted achromatic region, The gain setting unit sets the white balance gain and position set for each class classified by the classification to the white balance gain and position of the neighboring achromatic color area. The information processing device according to claim 8 .

10. The gain setting unit divides the color polarization image into regions and sets the white balance gain for each divided region. The information processing device according to claim 6 .

11. The gain setting unit switches the setting of the white balance gain for the color polarization image in accordance with the variation in the white balance gain set for each achromatic region extracted by the achromatic region extraction unit. The information processing device according to claim 6 .

12. When the variation in the white balance gain is within a predetermined tolerance range, the gain setting unit sets a white balance gain to be used in the entire region of the color polarization image based on the white balance gain set for each achromatic region. The information processing device according to claim 11.

13. When the variation in the white balance gain exceeds a predetermined tolerance range, the gain setting unit sets a white balance gain for an area different from the achromatic area based on the white balance gain set for each of the achromatic areas. The information processing device according to claim 11.

14. Using polarization information obtained from a color polarization image, a linear polarization degree or a Stokes vector is calculated as achromatic color determination information for each color component of the color polarization image, and an achromatic color region extraction unit extracts an achromatic color region that satisfies the condition that the achromatic color determination information is achromatic; a gain setting unit sets a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the achromatic color region extracted by the achromatic color region extraction unit achromatic; An information processing method including:

15. A program for causing a computer to perform white balance adjustment, comprising: a step of calculating a degree of linear polarization or a Stokes vector as achromatic color determination information for each color component of the color polarization image using polarization information acquired from the color polarization image, and extracting an achromatic color region that satisfies the conditions for the achromatic color determination information to be an achromatic color; a step of setting a white balance gain used in white balance adjustment of the color polarization image to a white balance gain that makes the extracted achromatic color area achromatic; A program for causing the computer to execute the above.

Images