IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM

Abstract

To provide an image processing device which can composite an image illuminated from an arbitrary direction using polarization information.SOLUTION: An image processing device (304) can execute: first processing of acquiring surface normal information of a subject; second processing of acquiring a plurality of photographed images obtained by imaging the subject while changing a polarization angle of light from the subject; third processing of acquiring the polarization information of the subject; and fourth processing of generating a composite image in which an illumination state of the subject is changed with respect to the photographed image. The third processing acquires, using the photographed image, light intensity information that changes according to the polarization angle. The fourth processing extracts a partial region of an image region of the photographed image or composite image using the surface normal information and generates the composite image using the partial region and the light intensity information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device that uses polarization information to change the illumination conditions of an image. [Background technology]

[0002] Conventionally, image processing devices that change the illumination state of an image using polarization information are known. Patent Document 1 discloses an image processing device that calculates an angle-dependent component at a specific polarization angle among angle-dependent components, which are luminance components that change according to the polarization angle, and generates an image using the angle-dependent component at the specific polarization angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-228910 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, since the polarization angle has a period of 180°, for example, when the polarization angle is 0° and 180°, or when the polarization angle is 90° and 270°, it is not possible to distinguish between these polarization angles. For this reason, the image processing device disclosed in Patent Document 1 cannot synthesize images illuminated from any direction.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an image processing device, an imaging device, an image processing method, and a program capable of synthesizing images illuminated from any direction using polarization information. [Means for solving the problem]

[0006] According to one aspect of the present invention, there is provided an image processing apparatus for a plurality of first images obtained by capturing images of the subject in a first illumination state at first and second viewpoints while changing a polarization angle of light from the A first process for obtaining Using the plurality of first images, light intensity information that changes according to the polarization angle is obtained, and polarization information of the subject is obtained based on the light intensity information. and a second operation to obtain generating a second image of the object corresponding to a second lighting condition different from the first lighting condition Third Processing and in the third process, a sign of a difference between the polarization information at the first viewpoint and the polarization information at the second viewpoint is used to performA partial region of the first image is extracted, and the second image is generated by changing a luminance value in the partial region based on the information on the second lighting condition and the light intensity information.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and a program capable of synthesizing images illuminated from any direction using polarization information. [Brief description of the drawings]

[0009] [Figure 1] 3 is an explanatory diagram of an imaging system and a polarizing element for acquiring polarization information in the first and second embodiments. FIG. [Diagram 2] FIG. 2 is an explanatory diagram of the axial arrangement of polarizing elements in the first and second embodiments. [Diagram 3] 3 is an explanatory diagram of an imaging system and a polarizing element for acquiring polarization information in the first and second embodiments. FIG. [Figure 4] 1 is an external view of an imaging device according to first and second embodiments. [Diagram 5] FIG. 1 is a block diagram of an imaging device according to first and second embodiments. [Figure 6] 4 is an explanatory diagram of a method for calculating an angle dependent component in the first and second embodiments. FIG. [Figure 7] 1 is an explanatory diagram of an angle-dependent component I(α) and an approximation function F(α) in Examples 1 and 2. FIG. [Figure 8] 3 is an explanatory diagram of a light source unit in the first and second embodiments. FIG. [Figure 9] 4 is an explanatory diagram of an imaging system and a polarization angle for acquiring polarization information in the first and second embodiments. FIG. [Figure 10] 4 is an explanatory diagram of a specular reflection component in the first and second embodiments. FIG. [Figure 11] 4 is an explanatory diagram of a surface normal vector in the first and second embodiments. FIG. [Figure 12]3 is an explanatory diagram of an imaging system and a polarizing element for acquiring polarization information in the first and second embodiments. FIG. [Figure 13] FIG. 13 is an explanatory diagram of the principle of area extraction in the third to sixth embodiments. [Figure 14] FIG. 2 is an explanatory diagram of a two-dimensional image sensor in accordance with Examples 3 to 6. [Figure 15] FIG. 1 is an explanatory diagram of a polarizing element in Examples 3 to 6. [Figure 16] FIG. 1 is an explanatory diagram of a polarizing element in Examples 3 to 6. [Figure 17] FIG. 13 is an explanatory diagram of a method for calculating the angle dependent component Id(α) in Examples 3 to 6. [Figure 18] FIG. 13 is an explanatory diagram of an approximation function F(α) in Examples 3 to 6. [Figure 19] FIG. 1 shows the results of Example 3. [Figure 20] FIG. 1 shows the results of Example 4. [Figure 21] FIG. 1 shows the results of Example 5. [Figure 22] FIG. 13 is a block diagram of an image processing system according to a sixth embodiment. [Figure 23] FIG. 13 is an explanatory diagram of an imaging system and polarization angles for acquiring polarization information in Examples 3 to 6. [Figure 24] FIG. 13 is a diagram showing the relationship between brightness and polarization angle in Examples 3 to 6. [Diagram 25] FIG. 1 is an explanatory diagram of the degree of polarization in Examples 3 to 6. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0011] First, there will be described matters common to Examples 1 and 2. In Examples 1 and 2, the following image processing is performed using an input image generated by capturing a plurality of polarized lights having different polarization directions from an object.

[0012] In the image processing, the brightness value of the input image is used to separate the input image into an angle-dependent component, which is a brightness component that changes according to the polarization angle α that the polarization direction makes with respect to the reference axis, and an angle-independent component, which is a brightness component that does not change even if the polarization angle α changes, and the angle-dependent component at a specific polarization angle is calculated. In addition, the surface normal information of the subject is separately obtained, and a partial area of ​​the image is extracted using the information, and an output image is generated for the extracted area using the angle-dependent component at a specific polarization angle, thereby obtaining an output image in which the illumination direction for the object is arbitrarily and intuitively adjusted.

[0013] Here, the definition of the polarization angle α in the first and second embodiments will be described with reference to Figs. 9(a) and (b). Figs. 9(a) and (b) are explanatory diagrams of an imaging system and polarization angle for acquiring polarization information. Fig. 9(a) shows the configuration of an imaging system (e.g., a digital camera) that captures images to acquire an input image, and the relationship with coordinate axes. The imaging system is composed of a polarizing plate 2, an optical system 1, and an imaging element 3.

[0014] The direction in which the optical axis OA of optical system 1 extends is defined as the z-axis, and two axes that are orthogonal to each other in a plane orthogonal to the z-axis are defined as the x-axis and y-axis. As shown in Fig. 9(b), when polarized light PL transmitted through polarizing plate 2 having a transmission axis in the xy plane is imaged by image sensor 3, the angle that the transmitted polarization direction makes with the reference axis (x-axis) is called the polarization angle α of the polarized light. The polarization angle α is an angle in the range of 0°≦α<180°.

[0015] The principle of adjusting the illumination direction using the angle-dependent component, which is a luminance component that changes according to the polarization angle α, will be described below. When the luminance value of each pixel in an image captured by capturing multiple polarized lights with different polarization angles α is plotted against the polarization angle α, the luminance value at each pixel is expressed by the following formula (1).

[0016] I=I c +I(α) …(1) In equation (1), I(α) is the angle-dependent component as the luminance component that changes with the polarization angle α, and I cis an angle-dependent component as a luminance component that does not change depending on the polarization angle α. In the image processing of this embodiment, the angle-dependent component I(α) and the angle-dependent component I c These are considered to be the specular reflection component and the diffuse reflection component from the object, respectively. This makes use of the fact that the specular reflection component reflects according to Fresnel's law, so contains more s-polarized components than p-polarized components and shows dependence on the polarization angle, whereas the diffuse reflection component does not depend on polarization.

[0017] The specular reflection component when illuminating from a specific direction α0 can be approximated by the angle-dependent component I(α1), α1 = α0 + 90 (deg). The image generated by imaging an object illuminated from the direction α0 is (Diffuse reflection component) + Specular reflection component when illuminated from the direction α0 Therefore, using the above approximation, the luminance value I' of each pixel when the object is illuminated from the direction α0 can be obtained by the angle independent component I C Using the angle-dependent component I(α0+90) at a specific angle, the luminance value I' can be expressed as in the following equation (2). c and the angle-dependent component I(α) at ​​a specific angle.

[0018] I'=I c +I(α0+90) …(2) Therefore, by calculating the luminance I' of each pixel from equation (2), it is possible to synthesize an image equivalent to when the object is illuminated from the direction α0.

[0019] In addition, equation (2) can be written as c , k, it may be expressed as the following equation (3).

[0020] I'=k c I c +k·I(α0+90) …(3) In this case, k c By varying and k, the ratio of specular reflection components to diffuse reflection components can be adjusted.

[0021] However, as mentioned above, the polarization angle α is in the range of 0°≦α<180°, and I(α0) when α0=0° and α0=180° are equivalent. Therefore, in reality, I'=I c The image created by +I(0°+90°) contains specular reflection components from both the 0° and 180° directions. Therefore, this embodiment uses surface normal information to resolve this ambiguity of 180° and obtain an image illuminated only from the 0° or 180° direction.

[0022] Specifically, only a portion of the image area is extracted using surface normal information, and image synthesis is performed using angle-dependent components at a specific angle only for the extracted portion. For example, when creating an image when illuminated from a 0° direction, surface normal information is used to extract an area that satisfies 0° or more and less than 90° or 270° or more and less than 360° (i.e., α-90° or more and less than α+90° with respect to the direction α to be illuminated). The extracted area does not include an area that is given specular reflection when illuminated from the α0+180° direction, so an image illuminated only from the 0° direction can be created by creating an image using formula (2) or formula (3) only for this area. The surface normal information is information regarding the orientation of the surface of the subject, such as a surface normal vector.

[0023] Photometric stereo is described below as an example of a method for acquiring a surface normal vector. Photometric stereo is a method in which the reflectance characteristics of an object are assumed based on the object's surface normal and the direction from the object to the light source, and the surface normal is calculated from the assumed reflectance characteristics and the luminance information of the object at multiple light source positions. If the reflectance cannot be uniquely determined when a specific surface normal and light source position are given, the reflectance characteristics can be approximated by a Lambertian reflection model that follows Lambert's cosine law.

[0024] FIG. 10 is an explanatory diagram of the specular reflection component (Torrance-Sparrow model). As shown in FIG. 10, the specular reflection component depends on the angle δ between the bisector of the light source vector s and the line of sight vector v, and the surface normal n. Therefore, the reflection characteristic may be a characteristic based on the line of sight. In addition, the accuracy information may be obtained by capturing images of the subject when the light source is turned on and when it is turned off, and calculating the difference between these images to remove the influence of light sources other than those assumed, such as ambient light.

[0025] In the following, we will explain the case where the reflection characteristics are assumed to be based on the Lambertian reflection model. Let i be the luminance value of the reflected light, and ρ be the Lambertian diffuse reflectance of the object. d Let E be the intensity of the incident light, s be the unit vector (light source direction vector) indicating the direction from the object to the light source (light source direction), and n be the normal vector of the unit surface of the object. In this case, the luminance value i is expressed by the following equation (4) according to Lambert's cosine law.

[0026] i=Eρ d s·n …(4) Here, let each component of the M different light source vectors (M≧3) be s1, s2, . . . , s M , the luminance values ​​of each component of the light source vector are i1, i2, . . . , i M Then, equation (4) can be expressed as equation (5) below.

[0027]

number

[0028] In equation (5), the left side is the luminance vector with M rows and 1 column, and the right side is [s1 T , s M T ] is the incident light matrix S that indicates the light source direction with M rows and 3 columns, and n is the unit surface normal vector with 3 rows and 1 column. When M=3, the inverse matrix S of the incident light matrix S -1 By using Eρ d n is expressed as follows:

[0029]

number

[0030] The norm of the vector on the left side of equation (6) is the incident light intensity E and the Lambertian diffuse reflectance ρ d The normalized vector is calculated as the surface normal vector of the object. That is, the incident light intensity E and the Lambertian diffuse reflectance ρ d appears in the conditional expression only in the form of a product, so Eρ d Assuming that is one variable, equation (6) can be regarded as a simultaneous equation that determines three unknown variables together with the two degrees of freedom of the unit surface normal vector n. Therefore, each variable can be determined by acquiring precision information using at least three light sources. Note that, if the incident light matrix S is not a regular matrix, an inverse matrix does not exist, so it is necessary to select each of the components s1 to s3 of the incident light matrix S so that the incident light matrix S becomes a regular matrix. In other words, it is preferable to select the component s3 to be linearly independent of s1 and s2.

[0031] When M>3, the number of conditional equations obtained is greater than the number of unknowns to be determined. Therefore, the unit surface normal vector n can be calculated from three arbitrarily selected conditional equations in the same manner as when M=3. When four or more conditional equations are used, the incident light matrix S is no longer a regular matrix, so an approximate field can be calculated using, for example, the Moore-Penrose pseudoinverse matrix. The unit surface normal vector n can also be calculated using a fitting method or an optimization method.

[0032] If the unit surface normal vector n is calculated using a luminance value that cannot be accurately obtained due to shading or luminance saturation among the luminance values ​​of each component of the light source vector, it becomes difficult to calculate an accurate normal vector. Therefore, the unit surface normal vector n may be calculated without using the luminance value that cannot be accurately obtained due to shading or luminance saturation. In other words, the light source vector s of M=m m The brightness value i obtained by m If is shaded or saturated, the light source vector s m and the brightness value i mis excluded from equation (3) to calculate the unit surface normal vector n. The luminance values ​​to be excluded may be determined by judgment based on a predetermined threshold. However, as described above, at least three pieces of luminance information are required. If the reflection characteristics of the subject are assumed to be a model other than the Lambertian reflection model, the conditional equation may differ from the linear equation for each component of the unit surface normal vector n. In this case, if a conditional equation with more than the unknown variables is obtained, a fitting method or optimization method can be used.

[0033] Furthermore, when M>3, multiple conditional expressions between 3 and M-1 can be obtained, so multiple solution candidates for the unit surface normal vector n can be obtained. In this case, a solution can be selected from the multiple solution candidates using another condition. For example, the continuity of the unit surface normal vector n can be used as a condition. When calculating the unit surface normal n for each pixel of the imaging device, if the surface normal at pixel (x, y) is n(x, y), and n(x-1, y) is known, a solution that minimizes the evaluation function expressed by the following formula (7) can be selected.

[0034]

number

[0035] In addition, if n(x+1, y) or n(x, y±1) is known, then the solution that minimizes the following equation (8) can be selected.

[0036]

number

[0037] If there are no known surface normals and there is uncertainty about the surface normals at all pixel positions, then a solution may be chosen that minimizes the sum of equation (8) over all pixels, as shown in equation (9) below.

[0038]

number

[0039] It is also possible to use the surface normal at a pixel other than the nearest pixel, or to use an evaluation function weighted according to the distance from the pixel position of interest. Also, as another condition, luminance information at an arbitrary light source position may be used. In a diffuse reflection model represented by the Lambertian reflection model, the closer the unit surface normal vector is to the light source direction vector, the greater the luminance of the reflected light. Therefore, the unit surface normal vector can be determined by selecting a solution that is closest to the light source direction vector with the largest luminance value among the luminance values ​​in multiple light source directions.

[0040] In addition, in the specular reflection model, if the light source vector is s and the unit vector in the direction from the object to the camera (the camera's line of sight vector) is v, the following equation (10) holds.

[0041]

number

[0042] As shown in equation (10), if the light source direction vector s and the camera line of sight vector v are known, the unit surface normal vector n can be calculated. If the surface is rough, the specular reflection also has a spread in the exit angle, but since it spreads near the solution obtained for a smooth surface, it is sufficient to select the candidate solution that is closest to the solution for a smooth surface from among multiple solution candidates. The true solution may also be determined by averaging the directions of multiple solutions.

[0043] The method of acquiring a surface normal using photometric stereo has been described above. However, in each embodiment, other information may be used instead of the surface normal vector. For example, information on the position of projection of the surface normal of the subject onto a plane (the xy plane of the surface normal vector, i.e., the imaging surface of the imaging element or a plane perpendicular to the optical axis OA) can be acquired and used as surface normal information. Here, the information on the projection position is, for example, information on which of the first to fourth quadrants based on the optical axis OA the subject is included in.

[0044] As shown in Fig. 11, the surface normal vector is expressed by an azimuth angle φ1 (0°≦φ1<360°) and a zenith angle φ2 (0°≦φ1<180°). In this case, only information is obtained to determine whether condition 2, where φ1 satisfies 0°≦φ1<180° or where 180°≦φ1<360°, is satisfied. This information indicates whether the surface normal is included in the first or second quadrant or the third or fourth quadrant, and therefore the information can be used as surface normal information.

[0045] In each embodiment, surface normal information can be obtained based on a plurality of images obtained by photographing a subject while sequentially irradiating the subject with light from a plurality of light sources at different positions. Also, surface normal information can be obtained based on regions with different luminance in a plurality of images obtained by photographing a subject while sequentially irradiating the subject with light from a plurality of light sources at different positions. EXAMPLES

[0046] First, the image processing procedure (image processing method) in the first embodiment of the present invention will be described in detail. The image processing method in this embodiment is composed of four steps, namely, (1) an image acquisition step, (2) a surface normal information acquisition step, (3) a polarization information acquisition step, and (4) an image synthesis step.

[0047] (1) Image acquisition step In the image acquisition step, a captured image is acquired. The captured image is a digital image obtained by capturing light that has passed through the imaging optical system and the polarizing element with the imaging element. The polarizing element is an element that can control the direction (angle) of the transmission axis of the transmitted polarized light. The direction of the transmission axis corresponds to the direction of the polarization direction, that is, the polarization angle α.

[0048] As the simplest polarizing element, a polarizing plate 2 shown in FIG. 12 can be used. FIG. 12 is an explanatory diagram of an imaging system and a polarizing element for acquiring polarization information in this embodiment, and shows an example of an element configuration for capturing an image in this embodiment. In addition to the polarizing plate 2, an optical system 1 and an imaging element 3 are provided. Here, the direction along the optical axis OA of the optical system 1 (optical axis direction) is the z axis, and the directions perpendicular to each other in a plane perpendicular to the z axis are the x axis and the y axis. In this case, the polarizing plate 2 is arranged in the xy plane and rotated in the xy plane around the z axis, so that the direction of the transmitted polarized light (polarization angle α) can be controlled.

[0049] Alternatively, a polarizing element 20 composed of a plurality of elements as shown in Fig. 1 can be used as another polarizing element. Fig. 1 is an explanatory diagram of an imaging system and a polarizing element for acquiring polarization information as a modified example of this embodiment. The polarizing element 20 is composed of a 1 / 4 λ plate 21, a liquid crystal variable phase plate 22, and a polarizing plate 23.

[0050] FIG. 2 is an explanatory diagram of the axial arrangement of the polarizing element 20, and the axial directions of the 1 / 4λ plate 21, the liquid crystal variable phase plate 22, and the polarizing plate 23 constituting the polarizing element 20 are indicated by dashed arrows. The axial directions indicate the slow axis for the 1 / 4λ plate 21 and the liquid crystal variable phase plate 22, and the transmission axis for the polarizing plate 23. The liquid crystal variable phase plate 22 can change the phase difference that the liquid crystal variable phase plate 22 gives to the incident light according to the applied voltage. The polarizing element 20 can change the direction of the polarized light that passes through the entire polarizing element 20 (i.e., the direction of the transmission axis) by controlling the applied voltage to the liquid crystal variable phase plate 22, and can obtain the same effect as rotating the polarizing plate 2 shown in FIG. 12. The polarizing element 20 controls the polarization angle α by the applied voltage, and therefore can be driven at a higher speed than by rotating the polarizing plate 2, and can capture a plurality of input images at different polarization angles α in a shorter time.

[0051] As yet another polarizing element, a polarizing element 30 as a polarizer array in which a plurality of polarizers are arranged in the xy plane can be used as shown in Figures 3(a) and (b). Figures 3(a) and (b) are explanatory diagrams of an imaging system and a polarizing element for acquiring polarization information as yet another modified example of this embodiment.

[0052] As shown in FIG. 3(b), for example, the multiple polarizers are arranged in a multiple number of sets, each set consisting of four polarizers with different transmission axis directions (indicated by dashed arrows). The polarizing element 30 is provided integrally with the image sensor 3 or disposed in its vicinity, and enables imaging of polarized light with different polarization angles α for each pixel of the image sensor 3 or for each pixel area including multiple pixels. An input image obtained by this imaging is an image containing luminance information of polarized light with different polarization angles α for each pixel or each pixel area. In other words, luminance information of multiple polarized lights can be obtained from a single input image.

[0053] In this embodiment, one or multiple captured images are obtained by capturing multiple polarized lights with different polarization angles α. When capturing captured images, it is preferable to store multiple input images with different polarization angles α as one file or to store them in association with each other using Exif information.

[0054] The method of acquiring the captured image is not particularly limited, but in the case where an image processing device that performs the above-mentioned image processing is built into an imaging device including an imaging element, the image processing device may acquire the image directly within the imaging device. In addition, in the case where the image processing device is provided separately from the imaging device, the image processing device may be connected to the imaging device via wired or wireless communication, and the imaging device may acquire the image from the image processing device via communication. In addition, the imaging device may store the image in a recording medium such as a memory card, and the image processing device may acquire the image via the recording medium.

[0055] (2) Surface normal information acquisition step In the surface normal acquisition step, an image for acquiring surface normal information is captured, and the surface normal information is calculated from the acquired image. In this embodiment, the surface normal is acquired using photometric stereo. The imaging device of this embodiment will be described with reference to Figs. 4(a) to (c) and Fig. 5. Fig. 4(c) is an external view of the imaging device 100 in this embodiment, and Fig. 5 is a block diagram of the imaging device 100.

[0056] As shown in FIG. 4(c), the imaging device 100 has an imaging section 300 (FIG. 4(b)) that images a subject and a light source section 400 (FIG. 4(a)). In this embodiment, the light source section has eight light sources 400a to 400h, but is not limited to this. At least three light sources are required to perform photometric stereo, so at least three light sources may be provided to acquire an input image. In this embodiment, eight light sources are arranged at equal intervals in a concentric manner at an equal distance from the optical axis OA of the imaging optical system that constitutes the imaging section 300, but is not limited to this. In this embodiment, the light source section 400 is detachably attached to the imaging section 300, but is not limited to this. The light source section 400 may be configured integrally with the imaging section 300.

[0057] The imaging unit 300 has an imaging optical system 301 and an imaging element 302. The imaging optical system 301 has an aperture 301a, and forms an image of light from a subject on the imaging element 302. The imaging element 302 is composed of a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and captures an image of the subject. That is, the imaging element 302 photoelectrically converts an image (optical image) of the subject formed by the imaging optical system 301, and generates an analog electrical signal (image data corresponding to an input image). The A / D converter 303 converts the analog signal generated by the photoelectric conversion of the imaging element 302 into a digital signal, and outputs the digital signal to an image processing unit 304.

[0058] The image processing unit (image processing device) 304 performs various image processing on the digital signal input from the A / D converter 303. The image processing unit 304 also calculates normal information of the subject. The image processing unit 304 has an input image acquisition unit 304a and a normal information acquisition unit 304b. The normal information output by the image processing unit 304 is stored in a recording unit 309 such as a semiconductor memory or an optical disk. The output information may also be displayed on a display unit (display) 305. Note that in this embodiment, the input image acquisition unit 304a and the normal information acquisition unit 304b are built into the imaging device 100, but may be provided separately from the imaging device 100.

[0059] The information input unit 308 supplies the system controller 310 with the shooting conditions (aperture value, exposure time, focal length, etc.) selected by the user. The imaging control unit 307 acquires an image under the desired shooting conditions selected by the user based on the information from the system controller 310. The irradiation light source control unit 306 controls the light emission state of the light source unit 400 in response to a control instruction from the system controller 310. Note that in this embodiment, the imaging optical system 301 is configured integrally with the imaging device, but is not limited to this. The present invention is also applicable to a camera system such as a single-lens camera configured with an imaging device body having an imaging element and an imaging optical system (interchangeable lens) that is detachable from the imaging device body.

[0060] Next, the surface normal acquisition process in this embodiment will be described. The surface normal acquisition process in this embodiment is executed by the system controller 310 and the image processing unit 304 according to a processing program as a computer program. The processing program is stored in, for example, a computer-readable storage medium (such as an internal memory of the system controller 310).

[0061] First, the input image acquisition unit 304a acquires a plurality of input images acquired by the imaging unit 300 by capturing images of a subject at a plurality of light source positions that are different from each other. The plurality of input images can be acquired by sequentially irradiating light from a single light source while changing the position of the single light source (using a driving unit or the like). Alternatively, the plurality of input images may be acquired by sequentially irradiating light from a plurality of light sources (e.g., eight light sources 400a to 400h shown in FIG. 4(a)) that are respectively at different positions.

[0062] Next, the normal information acquisition unit 304b acquires normal information. In this embodiment, specifically, the normal information acquisition unit 304b acquires normal information of the subject using a plurality of input images acquired by capturing images of the subject at a plurality of light source positions that are different from each other. The normal information is calculated based on the change in luminance information due to the light source position using photometric stereo. (3) Polarization information acquisition step In the polarization information acquisition step, the brightness value I of each pixel is extracted from the input image obtained in the image acquisition step, and the angle-dependent component I(α) is calculated from the brightness value I, and the angle-independent component I c Calculate the angle independent component I c may be calculated together with the angle-dependent component I(α), but this is not necessary. c If it is not possible to calculate directly, it can be calculated indirectly using the luminance value I and equation (1).

[0063] A method for calculating the angle-dependent component I(α) will be described below. The angle-dependent component I(α) is a luminance component that varies according to the polarization angle α, and can therefore be expressed as a sine function or cosine function with a period of 180°. For example, if the maximum value of the angle-dependent component I(α) is A and the polarization angle at which the angle-dependent component I(α) reaches the maximum value A is θ, the angle-dependent component I(α) is expressed by the following equation (11).

[0064] I(α)=Acos 2 (α-θ) …(11) In addition, the luminance value I of each pixel is divided into angle-independent components I cUsing this, it can be expressed as the following equation (12).

[0065] I=Acos 2 (α―θ)+I c …(12) By calculating A and θ from images captured at three or more different polarization angles α using equation (11) or equation (12), it is possible to obtain light intensity information that changes depending on the polarization angle α, i.e., the angle-dependent component I(α).

[0066] Here, a method for calculating the angle-dependent component I(α) will be described with reference to Figs. 6(a) and (b). Figs. 6(a) and (b) are explanatory diagrams of a method for calculating the angle-dependent component I(α). Figs. 6(a) and (b) show an example in which captured images are acquired at four different polarization angles α1, α2, α3, and α4. When the luminance values ​​I(α1) to I(α4) of corresponding pixels in the four input images are plotted against the polarization angle α, they can be expressed as shown in Fig. 6(a). Fig. 6(b) shows the result of fitting the luminance values ​​I(α1) to I(α4) shown in Fig. 6(a) using equation (12). The dotted circles in Fig. 6(b) represent the luminance values ​​I(α1) to I(α4) obtained from the four captured images, and the solid curve represents the fitting result. When the number of captured images is three, A, I c In addition, when there are four or more sheets, A, I can be calculated uniquely by using an optimization method such as the least squares method. c , can be calculated. (4) Image synthesis step In the image synthesis step, a portion of the image area is extracted using the surface normal information acquired in the surface normal information calculation step, and image synthesis is performed on the extracted portion using the angle information I(α) calculated in the polarization information acquisition step to generate an output image.

[0067] As described above, the specular reflection component I'(α0) when the object is illuminated from the direction α0 and the direction α0+180° can be approximated to the angle-dependent component I(α0+90) at α0+90(deg). However, the directions α0 and α0+180° cannot be distinguished from each other using only polarization information. Therefore, the surface normal information acquired in the surface normal information acquisition step is used. Specifically, the normal vector n is used to extract an area where the azimuth angle φ1 of the surface normal satisfies α0-90°≦φ1<α0+90°, and an image is synthesized using the angle-dependent component I(α0+90) that can be regarded as a specular reflection component only for the extracted area. Specifically, the brightness of each pixel is calculated for the extracted area using the following formula (13), and the image is synthesized.

[0068] I'=k·I(α0+90)+k c I c …(13) In addition, k and k in the formula (13) c is an arbitrary constant.

[0069] On the other hand, for areas other than the extracted area, the images are synthesized using, for example, the following equation (14).

[0070] I'=k c I c …(14) By separating the extracted region from the other region in this way, the direction α0 and the direction α0+180 are separated, and an image illuminated only from the direction α0 can be synthesized.

[0071] In the above explanation, a method for creating an image illuminated from a certain direction α0 has been described, but this embodiment is not limited to this, and images illuminated from multiple directions may be created. When creating an image illuminated from multiple directions, an area is extracted for each direction, and an angle-dependent component I(α n +90°) or k n ·I(α n +90°) should be added. (However, k n is an arbitrary constant.) Also, the angle independent component I c or k cI c By adding all the regions together, an image illuminated from multiple directions can be obtained.

[0072] In the above explanation, the angle-dependent component I(α) expressed by formula (11) is used to approximate the specular reflection component, but this embodiment is not limited to this. The angle-dependent component I(α) may be approximated by another approximation function F(α) and I' may be calculated using the approximation function F(α). Figures 7(a) to (c) are explanatory diagrams of the angle-dependent component I(α) (broken line) and the approximation function F(α) (solid line). As shown in Figures 7(a) to (c), it is preferable to make the angle at which the approximation function F(α) is maximum coincide with the angle at which the angle-dependent component I(α) is maximum.

[0073] In this embodiment, both the angle-dependent component and the angle-independent component are used to synthesize the images, but an image may be created from only the angle-dependent component. Also, in this embodiment, the imaging device for acquiring the surface normal information and the polarized image for acquiring the polarization information are described separately, but this is not limited to this, and a common imaging device equipped with a light source unit and a polarizing element may be used. EXAMPLES

[0074] Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that area information in which luminance changes between different captured images is used as surface normal information in the "(2) surface normal information acquisition step", and in the method of extracting a partial area from the surface normal information in the "(4) image synthesis step". Other configurations and methods in this embodiment are similar to those in the first embodiment, so their description will be omitted.

[0075] In this embodiment, in "(2) Surface normal information acquisition step", similarly to the first embodiment, the imaging device 100 shown in Fig. 4(c) and Fig. 5 is used to sequentially irradiate the light sources 400a to 400h and capture a plurality of captured images by capturing an object. Then, two of the eight captured images are compared to extract an area where the luminance changes, and the area information is stored as surface normal information corresponding to the light source of the image used for comparison.

[0076] In addition, it is preferable that the two captured images to be compared are light sources arranged at the diagonal positions closest to the optical axis, for example, a combination of light source 400a and light source 400e, or light source 400c and light source 400g, etc. are good. In addition, when extracting an area where the luminance changes, it is preferable to extract only an area that becomes bright when an image is acquired with the corresponding light source. For example, when comparing images captured with light source 400a and light source 400e, an area where the image captured with light source 400a is brighter than the image captured with light source 400e is stored as surface normal information corresponding to light source 400a. In addition, an area where the image captured with light source 400e is brighter than the image captured with light source 400a is stored as surface normal information corresponding to light source 400e. In this way, surface normal information is acquired for each light source. In the above description, eight light sources are sequentially irradiated to acquire eight images, but it is not necessary to capture images corresponding to all light sources. In addition, it is not necessary to acquire surface normal information for all light sources, and only necessary information may be acquired according to the image to be created using polarization information and the shape of the subject.

[0077] Furthermore, when capturing an image, it is not necessary to sequentially turn on the light sources 400a to 400h one by one. It is sufficient that the image is captured under different light source conditions each time. For example, the subject may be captured using four light sources at once, such as light sources 400a to 400d and light sources 400b to 400c, each time capture is performed, or the number of light sources may be changed each time capture is performed.

[0078] In addition, the light source unit 400 is not limited to the configuration shown in Fig. 4(a). The number of light sources is also not limited to eight, and may be two or more. For example, a light source unit 410 equipped with light sources 410a and 410b as shown in Fig. 8 may be used.

[0079] In this embodiment, in "(4) Image synthesis step", a partial region is extracted based on the surface normal information obtained as described above. To create an image illuminated from the direction α0, a light source whose light source position is close to the direction α0 is selected from the light sources 400a to 400h of the captured image, and the surface normal information corresponding to that light source is extracted as a partial region.

[0080] In this embodiment, the surface normal information is calculated using photometric stereo technology, but this is not limited to this, and the surface normal information may also be calculated using distance information to the subject obtained by a distance information measuring device such as a LIDAR.

[0081] As described above, the image processing apparatuses of the first and second embodiments can execute the first, second, third, and fourth processes. The first process acquires surface normal information of the object. The second process acquires captured images captured with different polarization angles (plural captured images obtained by capturing an object while changing the polarization angle of light from the object). The third process acquires polarization information of the object. The fourth process generates a composite image in which the illumination state of the object is changed for the captured image. The third process acquires light intensity information (angle-dependent component I(α)) that changes according to the polarization angle using the captured image. The fourth process extracts a partial area of ​​the image area of ​​the captured image or the composite image using the surface normal information, and generates a composite image using the extracted partial area and the light intensity information.

[0082] Next, a description will be given of matters common to Examples 3 to 6. In Examples 3 to 6, the following image processing is performed on an input image acquired by capturing polarized light at a plurality of different polarization angles at each of a plurality of different viewpoints.

[0083] In the image processing, at least two viewpoints are selected from among the multiple different viewpoints, and polarization information is calculated for each pixel using the brightness at multiple different polarization angles at each of these viewpoints. Here, the polarization information is one or both of the polarization direction and the degree of polarization. Next, any two viewpoints from among the selected viewpoints are treated as one pair, and the difference in polarization information is calculated for each of one or more pairs. Then, a specific area of ​​the image is extracted using the sign (positive or negative) of the difference. However, an area where the difference is zero may be included in either the positive or negative side.

[0084] In addition, from the luminance of a plurality of different polarization angles at a certain viewpoint, an angle-dependent component, which is a luminance component that changes according to the polarization angle, and an angle-independent component, which is a luminance component that does not change according to the polarization angle, are separated. Then, an output image is generated by calculating an angle-dependent component of a desired specific polarization angle in the specific region extracted as described above using the separated angle-dependent components. As a result, an output image is obtained in which the illumination direction with respect to the object is arbitrarily and intuitively controlled.

[0085] Here, the definition of the polarization angle α in Examples 3 to 6 will be described with reference to Figs. 23(a) and (b). Figs. 23(a) and (b) are explanatory diagrams of an imaging system and polarization angle for acquiring polarization information. Fig. 23(a) shows the relationship between the configuration of an imaging system (e.g., a digital camera) that captures images to acquire an input image and the coordinate axes. The imaging system is composed of a polarizing plate (polarizing element) 501, an optical system 401, and a two-dimensional image sensor (imaging element) 310.

[0086] The optical axis OA of the optical system 401 is defined as the z-axis, and two axes perpendicular to each other in a plane perpendicular to the z-axis are defined as the x-axis and y-axis. As shown in FIG. 23(b), the angle that the polarization direction PL of the polarized light transmitted through the polarizing plate 501 having a transmission axis in a plane parallel to the xy plane makes with respect to the reference axis (x-axis) is called the polarization angle α. From FIG. 23(b), it can be seen that the polarization angle α has a period of 180°. Note that the polarization angle α is defined here with respect to the x-axis, but is not limited to this, and may be an angle with respect to the y-axis, or may be expressed in any coordinate system in which the polarization angle α is uniquely determined.

[0087] Next, the angle-dependent component of the luminance, which changes depending on the polarization angle α, and the angle-independent component, which does not change, as well as the polarization direction and degree of polarization, which are polarization information, will be described.

[0088] When the brightness I of each pixel in multiple input images (or one input image, as described later) obtained by capturing polarized light with multiple different polarization angles α from a certain viewpoint is plotted against the polarization angle α, the result is as shown in Figure 24. Figure 24 is a diagram showing the relationship between brightness I and polarization angle α, with the vertical axis representing brightness I and the horizontal axis representing polarization angle α. Therefore, the brightness I at each pixel can be calculated as shown in the following equation (15).

[0089] I(α)=Ic+Id(α) …(15) Here, Id(α) is the angle-dependent component, which is the luminance component that changes with the polarization angle α, and Ic is the angle-independent component, which is the luminance component that does not change with the polarization angle α. Here, the angle-dependent component Id(α) and the angle-independent component Ic are considered to be the specular reflection component and the scattered component from the object, respectively. This is because the specular reflection component occurs due to Fresnel reflection, and is thought to show polarization angle dependence because the s-polarized component is stronger than the p-polarized component, while the scattered component is thought to be depolarized during the scattering process and does not show polarization angle dependence.

[0090] The polarization direction is defined as the polarization angle when the brightness I is maximum. From the above, the polarization direction can be considered to be approximated to the direction of s-polarized light. If this polarization direction is φ and the maximum value of brightness I is Imax, Imax is expressed by the following equation (16).

[0091] Imax = Ic + Id(φ) … (16) It is possible to consider that the luminance I is minimum when the angle-dependent component Id(α) is zero. If the minimum value of I is Imin, then it is expressed by the following equation (17).

[0092] Imin = Ic …(17) The degree of polarization indicates the degree to which the light being imaged is polarized. It can be thought of as the degree to which the brightness contains an angle-dependent component. If the degree of polarization is ρ, it is defined by the following equation (18).

[0093] ρ=(Imax-Imin) / (Imax+Imin) …(18) [Principle of lighting direction control] Next, the principle of controlling the illumination direction using the angle-dependent component of polarized light will be explained. First, an image of an object when it is uniformly illuminated can be expressed by the following equation (19).

[0094] (Scattered component) + (Specular reflection component)…(19) For example, the more the specular reflection component is increased, the brighter the object appears to be illuminated. Next, an image of an object illuminated from a specific direction β (the angle with respect to the x-axis is β) can be expressed by the following equation (20). (Scattered component) + (Specular reflection component when illuminated from a specific direction β) ... (20) In this case, the portion of the object surface facing the specific direction β, that is, the surface whose surface normal direction (projection of the surface normal onto the xy plane) is at an angle closer to the specific direction β, becomes brighter.

[0095] The specular reflection component when an object is illuminated from a specific direction β can be approximated by the angle-dependent component at a specific polarization angle β+90°, i.e., Id(β+90°). This is because, in Fresnel reflection, the s-polarized component is a polarized component perpendicular to the incident plane including the surface normal, so its polarization angle can be approximated as being perpendicular to the direction of the surface normal. In other words, when the polarization angle α is β+90°, the closer the direction of the surface normal is to β, the more s-polarized components are included in the luminance component of the object surface, and the larger the angle-dependent component Id(α), i.e., the specular reflection component, the brighter it appears.

[0096] Using the above-mentioned method of expressing illumination of an object, the luminance I'(β) of each pixel when the object is illuminated from a specific direction β can be approximated by the following equation (21) using the angle independent component Ic and the angle dependent component I(β+90°) at a specific polarization angle β+90°. In other words, the approximate luminance I' can be approximated by the linear sum of the angle independent component and the angle dependent component at a specific polarization angle.

[0097] I'(β)=Ic+Id(β+90°) …(21) Therefore, by calculating the approximate value I'(β) of the luminance of each pixel from equation (21), it is possible to synthesize an image that corresponds to an object illuminated from a specific direction β. Furthermore, equation (7) may be expressed as the following equation (22) using coefficients kc and kd, which are any value equal to or greater than 0.

[0098] I'(β)=kc·Ic+kd·Id(β+90°) …(22) In this case, by changing kc and kd, the ratio of the specular reflection component to the scattered component can be controlled.

[0099] However, as described above, the polarization angle α has a period of 180°. Therefore, the luminance I and the angle-dependent component Id of each pixel are the same at the polarization angle α and α+180°. Therefore, in an image of an object illuminated from a specific direction β, which is generated using the luminance approximation I', the angles with respect to the x-axis are indistinguishable between β and β+180°. In other words, the image is as if the object were illuminated from both β and β+180°. Therefore, the present invention aims to eliminate this 180° uncertainty by using the sign of the difference between the polarization information (polarization direction or polarization degree) acquired from multiple different viewpoints, and to obtain an image in which illumination from the β direction and the β+180° direction are distinguished and controlled.

[0100] Specifically, a specific region of the object image is extracted using the sign of the difference in polarization information between different viewpoints at each point on the object surface. Then, by synthesizing the image using the angle-dependent component Id(β+90°) of a specific direction β only in the extracted region, an output image can be generated in which the object is illuminated by selecting the β direction and the β+180° direction.

[0101] [Principle of ambiguity resolution using polarization direction] Next, the principle of area extraction based on the sign of the difference in polarization orientations calculated from a plurality of different viewpoints will be described with reference to Fig. 13. Fig. 13 is an explanatory diagram of the principle of area extraction.

[0102] FIG. 13(a) shows an imaging method when images of an object 10 (a sphere is shown as an example) captured from two viewpoints, viewpoint 1 and viewpoint 2, are used as input images. Here, an example is considered in which the optical axes of the optical systems at both viewpoints are parallel. The optical axis of the optical system 11 at viewpoint 1 is taken as the z-axis, and the direction from the object toward the optical system 11 is taken as positive. The optical systems at both viewpoints are considered to be approximated to a single thin lens, and an xy plane perpendicular to the z-axis is taken at the position of the pupil of the optical system 11 at viewpoint P1. In other words, the intersection of the z-axis and the xy plane is the origin of the coordinates here, and is the coordinates of viewpoint P1. In FIG. 13(a), the xz plane is parallel to the paper surface.

[0103] In FIG. 13(a), consider an example in which the pupil of the optical system 12 at viewpoint P2 is also included in the xy plane. The intersection of the optical axis at viewpoint P2 and the xy plane (which gives the coordinates of viewpoint P2) is on the x-axis, and the direction of the x-axis from viewpoint P2 to viewpoint P1 is taken as positive. In other words, the x-coordinate of viewpoint P2 is negative. The y-axis is perpendicular to the paper surface, and the direction from the back to the front of the paper surface is taken as positive. Based on the above, the coordinates of viewpoint P1 are expressed as (0,0,0), and the coordinates of viewpoint P2 are expressed as (X,0,0). Note that the polarizing elements 13 and 14 are parallel to the xy plane, and the polarization angle α of the transmitted polarized light is defined as the angle with respect to the positive direction of the x-axis, as described above.

[0104] The coordinates of a point 15 on the object surface are (x, y, z), and the surface normal n at the point 15 is expressed by the following equation (23).

[0105]

number

[0106] Here, n is a unit vector. Next, consider a ray of light that is reflected at point 15 and travels toward viewpoint 1. If this is expressed as vector V1, it can be expressed as in the following equation (24).

[0107]

number

[0108] The s-polarized component is perpendicular to the plane formed by the surface normal n and the reflected light ray V1. Therefore, if the polarization direction of the s-polarized component is expressed as a three-dimensional vector s1, it can be expressed as the following equation (25).

[0109]

number

[0110] Similarly, if the light ray reflected at point 15 and traveling toward viewpoint 2 is represented by vector V2, it can be expressed as the following equation (26).

[0111]

number

[0112] The s-polarized component is perpendicular to the plane formed by the surface normal n and the reflected light ray V2. Therefore, the polarization direction of the s-polarized component can be expressed as the three-dimensional vector s2 as shown in the following equation (27).

[0113]

number

[0114] Incidentally, the polarization direction φ can be approximated as the polarization angle α when the luminance components contain the most s-polarized components. In other words, it is the polarization angle α when the polarization direction of the s-polarized component in the xy plane coincides with the direction of the transmission axis of the polarizing element. The polarization directions of the s-polarized component in the xy plane at viewpoints 1 and 2 can be approximated as the projections of three-dimensional vectors s1 and s2 onto the xy plane. Expressing these as two-dimensional vectors s1' and s2', respectively, gives the following equations (28) and (29), respectively.

[0115]

number

[0116]

number

[0117] Therefore, if the polarization directions at point 15 at viewpoints 1 and 2 are φ1 and φ2, respectively, then the following equation (30) holds for the sign of the difference between φ1 and φ2 from vectors s1' and s2' in the xy plane described above.

[0118]

number

[0119] Here, sgn is a sign function. If the rightmost side of equation (30) is further transformed, it becomes the following equation (31).

[0120]

number

[0121] In many cases when photographing an actual object, the distance between the optical system and the object is sufficient in the z-axis direction, and considering that the angle of view is not too large, y / z can be approximated as 0. Therefore, the following equation (32) holds for the sign of the difference between φ1 and φ2.

[0122]

number

[0123] 13(a), since X<0, z<0, nz>0, when ny>0, i.e., when the object surface faces upward, the polarization orientation difference φ2-φ1 is negative. When ny<0, i.e., when the object surface faces downward, the polarization orientation difference φ2-φ1 is positive. Conversely, by using the sign of the polarization orientation difference, it is possible to extract a specific area in the object image according to whether the object surface faces upward or downward.

[0124] FIG. 13(b) shows an example of region extraction performed on the sphere 10 in FIG. 13(a). In FIG. 13(b), black represents a region where the difference sign is negative, and white represents a region where the difference sign is positive. In FIG. 13(b), only two different viewpoints in the x-axis direction (called parallax in the x-axis direction) are used, but parallax in other directions, for example, in the y-axis direction, may also be used. This makes it possible to extract regions according to the left-right direction as well as the up-down direction of the object surface. First, consider viewpoint 3, which is in the negative direction on the y-axis with respect to viewpoint 1, that is, has coordinates (0, Y, 0) (however, Y<0). In this case, when nx>0, that is, when the object surface faces right, the difference in polarization orientation φ2-φ1 is positive. When nx<0, that is, when the object surface faces left, the difference in polarization orientation φ2-φ1 is negative.

[0125] Fig. 13(c) shows an example of area extraction using the sign of the difference in polarization orientation due to parallax in the y-axis direction, performed on the sphere 10 in Fig. 13(a). In Fig. 13(c), black represents areas where the sign of the difference is negative, and white represents areas where the sign of the difference is positive.

[0126] FIG. 13(d) shows an example in which both the area extraction using the parallax in the y-axis direction and the area extraction using the parallax in the x-axis direction described above are performed on the sphere 10 in FIG. 13(a). In FIG. 13(d), the signs of the area 10-1 in FIG. 13(b) and (c) are negative and positive, the signs of the area 10-2 are negative and negative, the signs of the area 10-3 are positive and negative, and the signs of the area 10-4 are positive and positive. In this way, by using parallax in multiple different directions, the number of areas divided in the object image increases, and smaller areas can be extracted. However, it should be noted that when three or more different viewpoints exist on one line, the results will be the same no matter which two viewpoints are used to extract the area. When using parallax in multiple directions, it is desirable to select a pair of directions in which the angle between the directions of parallax is as large as possible, for example, a pair of directions that is a right angle for two directions.

[0127] By combining the region extraction described above with illumination control using the angle-dependent component Id(β+90°) of the specific direction β, the ambiguity of 180° according to the orientation of the object surface is eliminated. As a result, it is possible to generate an image in which the object is illuminated more naturally.

[0128] [Principle of ambiguity resolution using the degree of polarization] The principle of region extraction based on the sign of the difference in the degree of polarization (degree of linear polarization: DOLP) calculated from a plurality of different viewpoints will be described with reference to FIG. 13(a) again.

[0129] Here, we consider an example in which the degree of polarization is calculated using input images of an object captured from two viewpoints, viewpoint 1 and viewpoint 2. The dot product of the surface normal n at point 15 (x, y, z) on the object surface and the light ray V1 reflected at point 15 toward viewpoint 1 is given by the following equation (33).

[0130]

number

[0131] Similarly, the dot product of the surface normal n at point 15 and the ray V2 that is reflected at point 15 and heads toward the viewpoint 2 is expressed by the following equation (34).

[0132]

number

[0133] Let the angles between the surface normal n and the reflected rays V1 and V2 be θ1 and θ2, respectively. In order for the reflected rays V1 and V2 to be imaged, it is considered that 0≦θ1≦90° and 0≦θ2≦90° are satisfied. Therefore, 0≦n·V1≦1 and 0≦n·V2≦1.

[0134] Meanwhile, the degree of polarization ρi (i is a subscript and is either 1 or 2) at point 15 at viewpoints 1 and 2 can be expressed as in the following equation (35).

[0135]

number

[0136] where m is the refractive index of the material on the surface of the object, and k corresponds to the proportion of the specular reflection component in the brightness component at point 15, and satisfies 0≦k≦1. Since m and k depend on the surface characteristics of the material at point 15, they can be considered as constants independent of the imaging viewpoint. When equation (21) is plotted, it appears as shown in Figure 25.

[0137] FIG. 25 is an explanatory diagram of the degree of polarization. In FIG. 25, the horizontal axis indicates the angle θi, and the vertical axis indicates the degree of polarization ρ. The angle θB in FIG. 25 corresponds to the Brewster angle of Fresnel reflection (i.e., specular reflection). From this, it can be seen that the degree of polarization ρi increases monotonically with respect to θi when 0°≦θi≦θB, while it decreases monotonically when θB≦θi≦90°. From the above, the following equation (36) holds for the sign of the difference between ρ1 and ρ2, using the sign function.

[0138]

number

[0139] Regarding the distances d1 and d2 from point 15 to viewpoints 1 and 2 on the rightmost side of equation (36), in many cases in actual photography, the optical system and the object are sufficiently separated in the z-axis direction, and d1 and d2 are approximately equal. Considering this, d1 can be approximated as d1 = d2. Therefore, by further transforming the rightmost side of equation (36), the following equation (37) can be obtained.

[0140]

number

[0141] To summarize the above, the following equation (38) holds for the sign of the difference between ρ1 and ρ2.

[0142]

number

[0143] More roughly speaking, when the optical system and the object are far enough apart in the z-axis direction that the angle of view is not too large, the reflected rays V1 and V2 mentioned above can be considered to be parallel to the z-axis. Therefore, from a qualitative point of view, the angle θi (i=1, 2) between the surface normal n and V1 or V2 in equation (24) can be rephrased as the angle θz (0° to 90°) between the surface normal n and the z-axis.

[0144] Now, considering that X<0 and d1>0 from the definition in Figure 13(a), the following conclusion can be drawn. For nx>0, that is, for an object plane facing right, the difference in the degree of polarization ρ2-ρ1 is positive when θz is less than θB and negative when it is greater than θB. For nx<0, that is, for an object plane facing left, the difference in the degree of polarization ρ2-ρ1 is negative when θz is less than θB and positive when it is greater than θB.

[0145] Conversely, by using the sign of the difference in the degree of polarization, a specific region in the object image can be extracted according to the left-right orientation of the object surface. In other words, the region is divided in a direction perpendicular to the up-down orientation when the sign of the difference in the polarization orientation is used. FIG. 13(e) shows an example of region extraction performed on the sphere 10 in FIG. 13(a). In FIG. 13(e), black represents a region where the sign of the difference is negative, and white represents a region where the sign of the difference is positive. As can be seen from FIG. 13(e), due to the existence of θB, the orientation of the left and right surfaces cannot be completely distinguished. With this in mind, it is better to use the sign of the difference in the degree of polarization for region extraction, taking into account the shape of the object.

[0146] In the above, only the parallax in the x-axis direction is used, but parallax in other directions, for example, the y-axis direction, may also be used. This makes it possible to extract areas according to the up-down direction as well as the left-right direction of the object surface. When using the polarization degrees calculated at viewpoint 1 and viewpoint 3 in the negative direction on the y-axis, the following is obtained. For ny>0, that is, for an upward-facing object surface, the polarization degree difference ρ2-ρ1 is positive when θz is less than θB and negative when it is greater than θB. For ny<0, that is, for a downward-facing object surface, the polarization degree difference ρ2-ρ1 is negative when θz is less than θB and positive when it is greater than θB.

[0147] Fig. 13(f) shows an example where the above-mentioned region extraction is performed on the sphere 10 of Fig. 13(a). In Fig. 13(f), black represents regions where the difference sign is negative, and white represents regions where the difference sign is positive. Furthermore, smaller regions may be extracted by using parallax in a plurality of different directions, such as by performing both region extraction using parallax in the y-axis direction and region extraction using parallax in the x-axis direction described above.

[0148] In the above, the area extraction was performed using the sign of the difference in only the polarization direction obtained by the parallax in one or more directions, or the sign of the difference in only the polarization degree. However, since both the polarization direction and the polarization degree are calculated from the input image, the area extraction may also be performed using the signs of the differences in both of these.

[0149] Fig. 13(g) shows an example of region extraction using parallax in the x-axis direction for the sphere 10 in Fig. 13(a). In Fig. 13(g), the signs of region 10-5 in Fig. 13(b) and Fig. 13(e) are negative and positive, those of region 10-6 are negative and negative, those of region 10-7 are positive and negative, and those of region 10-8 are positive and positive. In this way, when the parallax is in the same direction, the directions divided by the sign of the difference between the viewpoints are orthogonal for the polarization orientation and the degree of polarization, so by using both, it is possible to extract smaller regions in the object image.

[0150] By combining the region extraction described above with illumination control using the angle-dependent component Id(β+90°) of the specific direction β, the ambiguity of 180° according to the orientation of the object surface is eliminated. As a result, an image in which the object is illuminated more naturally can be created.

[0151] Next, the image processing procedure (image processing method) will be described. The image processing method is composed of four steps ST1 to ST4, namely, (1) image acquisition step ST1, (2) polarization information step ST2, (3) area extraction step ST3, and (4) image synthesis step ST4. These steps ST1 to ST4 also correspond to FIG. 22 of Example 4 described later.

[0152] (1) Image acquisition step ST1 In the image acquisition step ST1, an input image is acquired. The input image is a digital image obtained by capturing polarized light that has passed through an optical system and a polarizing element with a two-dimensional image sensor at each of a plurality of different viewpoints.

[0153] Multiple different viewpoints can be obtained by photographing an object from multiple different points (spatial coordinates) (example shown in Figure 13(a)). Multiple imaging devices at different points may be used, or one imaging device may be moved to multiple points. This method requires a process to match pixels showing the same position on the object between multiple input images. A typical example of this process is template matching, which estimates the amount of positional deviation between input images and aligns them so that this is zero.

[0154] In addition, multiple different viewpoints can also be obtained by dividing the pupil of the optical system. As a method of dividing the pupil, for example, it is possible to block a part of the aperture of the optical system asymmetrically with respect to the optical axis, and capture an image while successively changing the part that is blocked. As another example, in a two-dimensional image sensor, a method of effectively dividing the pupil can be considered by providing something (for example, a film) that blocks a part of the light incident on the pixel in a part close to the microlens or color filter that constitutes the pixel. As yet another example, it is possible to effectively divide the pupil by providing two or more photodiodes in each pixel of the two-dimensional image sensor.

[0155] 14(a) and (b) are explanatory diagrams of a two-dimensional image sensor, showing an example of a two-dimensional image sensor used for dividing the pupil of an optical system. FIG. 14(a) shows a two-dimensional image sensor 311a in which one pixel 31 has two photodiodes 31-1 and 31-2 arranged in the x-axis direction. This allows the pupil to be divided into two in the x-axis direction. FIG. 14(b) shows a two-dimensional image sensor 311b in which one pixel 32 has four photodiodes 32-1 to 32-4 arranged in both the x-axis direction and the y-axis direction. By using an input image generated from the luminance of each of the photodiodes 32-1 to 32-4, the pupil can be divided into two at an angle of 45° to the x-axis and two at an angle of 135° to the x-axis, for a total of four. In addition, by generating an input image using two brightnesses each of photodiodes 32-1 and 32-2, 32-2 and 32-3, 32-3 and 32-4, and 32-4 and 32-1, the pupil can be divided into two in the x-axis direction and two in the y-axis direction, for a total of four.

[0156] When obtaining multiple different viewpoints by dividing the pupil, it is sufficient to use one image capture device and capture an object from one point. Therefore, it is possible to omit the process of matching pixels showing the same position of the object between input images with different viewpoints. Therefore, operation and image processing are simpler than when capturing images from multiple positions.

[0157] A polarizing element limits the polarization direction (angle) of the transmitted light to the direction (angle) of the transmission axis of the polarizing element. In other words, the direction of the transmission axis corresponds to the polarization direction of the transmitted light, i.e., the polarization angle α. By making the direction of the transmission axis variable, it is possible to control the polarization angle α to a desired angle.

[0158] The simplest polarizing element that can be used is a polarizing plate 501 shown in Fig. 23. The polarizing plate 501 is arranged in a plane parallel to the xy plane, and by rotating it in the xy plane around the optical axis OA, the polarization angle α of the transmitted polarized light can be controlled.

[0159] 15(a) to (d) are explanatory diagrams of another polarizing element. As an example of another polarizing element, a polarizing element 50 composed of a plurality of elements can be used, as shown in FIG. 15(a). The polarizing element 50 is composed of a 1 / 4λ plate 51, a liquid crystal variable phase plate 52, and a polarizing plate 53.

[0160] 15(b) to (d), the axial directions of the 1 / 4λ plate 51, the liquid crystal variable phase plate 52, and the polarizing plate 53 constituting the polarizing element 50 are indicated by dashed arrows. The axial directions indicate the slow axis for the 1 / 4λ plate 51 and the liquid crystal variable phase plate 52, and the transmission axis for the polarizing plate 53. The liquid crystal variable phase plate 52 can change the phase difference that it gives to incident light in response to an applied voltage. The polarizing element 50 can change the polarization angle α of the polarized light that is transmitted through the entire polarizing element 50 by controlling the applied voltage to the liquid crystal variable phase plate 52, and can obtain the same effect as rotating the polarizing plate 501 shown in FIGS. 23(a) and (b).

[0161] 16(a) and (b) are explanatory diagrams of yet another polarizing element 402. The polarizing element 402 is a polarizer array in which a plurality of polarizers shown in FIG. 16(a) and (b) are arranged in a plane parallel to the xy plane. For example, as shown in FIG. 16(b), the plurality of polarizers are arranged in a plurality of sets of polarizers, each set consisting of four polarizers with different transmission axis directions (indicated by dashed arrows). The polarizing element 40 is provided integrally with the two-dimensional image sensor 311 (311a, 311b) or disposed in the vicinity thereof. Then, it is possible to capture polarized light with different polarization angles α for each pixel of the two-dimensional image sensor 311 (311a, 311b) or for each pixel area including a plurality of pixels. One input image obtained by this capture is an image containing luminance information of polarized light with different polarization angles α for each pixel or each pixel area. That is, luminance information of multiple polarized lights can be obtained from one input image.

[0162] As described above, in step ST1, multiple images or one image generated by capturing multiple polarized lights with different polarization angles α are acquired as input images. Note that, when acquiring the input images, it is preferable to store the multiple input images with different polarization angles α as one file or to store them in association with each other using Exif information.

[0163] The method of acquiring the input image is not particularly limited, but in the case where an image processing device that performs the above-mentioned image processing is built into an imaging device including a two-dimensional image sensor, the image processing device may acquire the input image directly within the imaging device. In addition, in the case where the image processing device is provided separately from the imaging device, they may be connected to each other so as to be able to communicate with each other by wire or wirelessly, and the imaging device may acquire the input image from the image processing device via communication. In addition, the imaging device may store an image in a recording medium such as a memory card, and the image processing device may acquire the input image via the recording medium.

[0164] (2) Ellipsometric analysis step ST2 In the polarization analysis step ST2 (polarization information acquisition step), the brightness I(α) at ​​each point on the object surface for each of the input images acquired in the image acquisition step ST1 is extracted for each of the different viewpoints. Then, the angle-dependent component Id(α) at ​​each point on the object surface is calculated from the brightness I(α), and the angle-independent component Ic is calculated as necessary. The angle-independent component Ic may be calculated together with the angle-dependent component Id(α), but this calculation is not necessarily required. When the angle-independent component Ic is not calculated directly, it can also be calculated indirectly using the brightness I and formula (15).

[0165] A method for calculating the angle-dependent component Id(α) will now be described. The angle-dependent component Id(α) is a luminance component that varies according to the polarization angle α with a periodicity of 180°, and can be expressed as a sine function or cosine function with a period of 180°. For example, if the maximum value of the angle-dependent component Id(α) is A and the polarization angle at this time (polarization direction from the above definition) is φ, the angle-dependent component Id(α) is expressed as the following formula (39).

[0166] Id(α)=A cos 2 (α-φ) …(39) Moreover, the luminance I of each pixel can be expressed as the following equation (40) by further using the angle independent component Ic.

[0167] I(α)=Ic+A cos 2 (α-φ) …(40) From equation (39) or equation (40), A and φ can be calculated from input images with three or more different polarization angles α, and the angle-dependent component Id(α) can be obtained.

[0168] 17(a) and (b) show an example of a method for calculating the angle-dependent component Id(α). These figures show an example of acquiring images with four different polarization angles α1, α2, α3, and α4 from a certain viewpoint among the input images. In these four images, when the brightness I(α1) to I(α4) of the corresponding pixels are plotted against the polarization angle α, they can be expressed as shown in FIG. 17(a). FIG. 17(b) shows the result of fitting the brightness I(α1) to I(α4) shown in FIG. 17(a) using equation (26). The dotted circles in FIG. 17(b) show the brightness values ​​I(α1) to I(α4) obtained from the four captured images, and the solid lines show the fitting curves. In this way, when there are four or more images with different polarization angles, A, φ, and Ic can be calculated using an optimization method such as the least squares method. When there are three images with different polarization angles, A, φ, and Ic can be uniquely calculated from equation (26).

[0169] From A, Ic, and φ calculated as described above, Imax and Imin shown in equations (16) and (17) can be found, respectively, and the degree of polarization ρ shown in equation (18) can be calculated. In this manner, A, Ic, φ, and ρ are calculated for each different viewpoint, and φ and / or ρ are output to region extraction step ST3, and A and φ (i.e., function Id(α)) and Ic, if necessary, are output to image synthesis step ST4.

[0170] (3) Region extraction step ST3 In the region extraction step ST3, the polarization information for each different viewpoint calculated in the polarization analysis step ST2, i.e., either or both of the polarization direction φ and the degree of polarization ρ, are obtained. Then, any two viewpoints among the multiple viewpoints are grouped into one pair, and one or more different pairs are created. In each pair, the differences between the viewpoints in φ and ρ are calculated for pixels corresponding to the same position on the object, to obtain a difference map. Then, a specific region is extracted using the code of the difference map.

[0171] Note that before region extraction, the difference map may be subjected to filtering using an averaging filter, a median filter, a Gaussian filter, etc. In region extraction, the region may be extracted by simply selecting the positive or negative of the difference map (when zero exists, zero may or may not be included), or the absolute value of the difference may be used as an index of reliability, and positive or negative regions having an absolute value equal to or greater than a certain threshold may be extracted.

[0172] Furthermore, when the extracted region is expressed by binarizing the image region to be extracted with a value of 1 and the other image regions with a value of 0, the binarized map may be subjected to a filter process such as an averaging filter or a Gaussian filter. Then, in the image synthesis step ST4 described later, the binarized map (having values ​​of 0 to 1) subjected to the filter process is used as a mask for the angle-dependent components in a specific direction β (before the 180° ambiguity is resolved) (a direct product is taken). This process produces the same effect as region extraction.

[0173] When extracting a region using a plurality of sets of different viewpoints, for example, the logical product of the regions extracted in each set may be taken, or a direct product may be taken when using as a mask having a value between 0 and 1. The specific region of the image of the object extracted in the above manner is output to image synthesis step ST4.

[0174] (4) Image synthesis step ST4 In the image synthesis step ST4, the specific area of ​​the image of the object extracted in the area extraction step ST3, the angle-dependent component Id calculated in the polarization analysis step ST2, and, if necessary, the angle-independent component Ic are obtained, and these are used to perform image synthesis to generate an output image.

[0175] As described above, the luminance of the angle-dependent component when an object is illuminated from a specific direction β (which cannot be distinguished from β+180°) is approximated by the angle-dependent component Id(β+90°) at a specific polarization angle β+90°. Here, the approximate value of the luminance of the angle-dependent component in which the 180° ambiguity of β is resolved is defined as Id'. Id' can be obtained by calculating Id(β+90°) only for pixels included in the extracted region, and can therefore be expressed as the following equation (41).

[0176] Id'(β)=Id(β+90°)·q …(41) where Id(α)=A cos 2 (α-φ), q is 1 if the pixel is included in the extraction region, and 0 if it is not. However, it is 0 to 1 when filtering is performed after binarization.

[0177] Therefore, an image of an object illuminated from a specific direction β (with the 180° ambiguity resolved) can be synthesized by calculating the luminance approximation I'(β) using the angle-independent component Ic according to the following equation (42).

[0178] I'(β)=kc·Ic+kd·Id'(β) =kc·Ic+kd·Id(β+90°)·q …(42) where kc and kd are arbitrary constants greater than or equal to 0 that control the ratio of the scattered component to the specular reflection component. Id(α) and q are the same as in equation (41).

[0179] In the above description, images illuminated from one specific direction β are synthesized, but the present invention is not limited to this, and images illuminated from multiple specific directions may be synthesized. In order to obtain an approximate value Id' of the brightness of the angle-dependent component in multiple specific directions βj (j is a subscript representing the number of the specific direction) (wherein the 180° ambiguity is eliminated), first, an extraction region is prepared for each of the specific directions βj. Next, in the prepared extraction region, the angle-dependent component Id(βj+90°) or kj·Id(βj+90°) at the specific polarization angle βj+90° corresponding to the specific direction βj is calculated. (Note that kj is an arbitrary constant equal to or greater than 0.) Finally, these are added together. This can be expressed as the following equation (43).

[0180]

number

[0181] where Id(α)=A cos 2 (α-φ), q j If the pixel is included in the extraction area for the specific direction βj, it is 1, and if not, it is 0. However, if filtering is performed after binarization, it is 0 to 1.

[0182] In addition, when an extraction area is commonly used in multiple specific directions (for example, all of the specific directions are illuminated from above the object, and therefore only the area of ​​the upward-facing surface in the object image needs to be identified), it is not necessary to repeat the area extraction process for all specific directions.

[0183] Furthermore, by adding the angle-independent components Ic or kc·Ic to the entire region, it is possible to synthesize images of an object illuminated from multiple specific directions βj (with the 180° ambiguity resolved). The approximation I' of the luminance of the object surface in this case is calculated by the following equation (44).

[0184]

number

[0185] where Id(α) and q jis the same as equation (43).

[0186] In the explanation so far, the angle-dependent component Id(α) expressed by the trigonometric function of equation (39) is used to calculate the approximate value Id' of the angle-dependent component brightness in a specific direction. However, this embodiment is not limited to this, and another approximation function F(α) may be used to approximate Id', and the approximate value I' of the brightness of each pixel may be calculated using this. In this case, Id' or I' is calculated by multiplying the angle-dependent component Id(α) in equations (41) to (44) by A·cos 2 It can be calculated by substituting F(α) for (α-φ). Any function can be used as the approximation function F(α), but it is preferable that the angle at which F(α) is maximized coincides with the polarization direction φ. Here, "coincide" may include some error, and a difference of several percent to several tens of percent is permitted.

[0187] 18(a)-(c) are explanatory diagrams of the approximation function F(α). In these figures, the dashed curve indicates Id(α) in FIG. 17(b), and the solid curve indicates F(α). As shown in these figures, by using an arbitrary F(α), the change in luminance value with respect to the polarization angle α can be made different from Id(α). Therefore, by using F(α) as the approximation value Id' of the luminance of the angle-dependent component in a specific direction, it becomes possible to control the illuminated area (illumination area or illumination width) of the object surface in the composite image and the illumination luminance distribution.

[0188] In the above description, both the angle-dependent component and the angle-independent component are used to synthesize the images, but an image may be created from only the angle-dependent component. EXAMPLES

[0189] Next, a third embodiment of the present invention will be described with reference to Figs. 19(a) to (f). Figs. 19(a) to (f) are diagrams showing the results of this embodiment. In this embodiment, from six input images obtained by capturing polarized light with polarization angles α of 0°, 45°, and 90° at each of two different viewpoints, an output image is generated that eliminates the 180° ambiguity of the illumination on the object by utilizing the sign of the difference in polarization orientation between the viewpoints. The object is an octahedral die shown in Fig. 19(a) (the binocular image when α=0°, which will be described below).

[0190] The two viewpoints in this embodiment are different in the horizontal direction. These two different viewpoints are obtained by dividing the pupil of the optical system into two in the horizontal direction. As in the example of FIG. 14(a), the pupil is effectively divided into two by capturing an image using a two-dimensional image sensor 311a in which each pixel has two photodiodes arranged in the horizontal direction (in most cases, the longitudinal direction of the two-dimensional image sensor).

[0191] By dividing the pupil, a captured image that shows an object as if seen from the right side is called a right-eye image, and a captured image that shows an object as if seen from the left side is called a left-eye image. By taking the sum or average of the luminance of the right-eye image and the left-eye image, a general captured image without dividing the pupil can be generated. This is called a two-eye image below. In principle, the blur caused by defocus in the right-eye image and the left-eye image is asymmetric in the pupil division direction (horizontal in this case). Therefore, in order to obtain a more natural output image, region extraction is performed using the right-eye image and the left-eye image, but it is desirable to use the angle-dependent component and angle-independent component calculated from the both-eye image in image synthesis that controls the lighting.

[0192] Figure 19(b) shows the result of calculating A, φ, and Ic in equation (40) using both-eye images with polarization angles α of 0°, 45°, and 90°, and synthesizing an image in which the object is illuminated from a specific direction β=45° with kc set to 1 and kd set to 1.3 in equation (42). It can be seen that due to the 180° period of the polarization angle, the directions of 45° and 225° with respect to the x-axis cannot be distinguished, and both the upper right and lower left faces are illuminated in the image.

[0193] Figures 19(c) and (e) show the results of region extraction using the sign of the difference in polarization orientation calculated for the left-eye image and the right-eye image. In Figure 19(c), the difference sign is negative, and in Figure 19(e), the positive sign is shown in white. As mentioned above, it can be seen that by selecting a region with a negative sign, it is possible to extract parts where the surface faces upward, and by selecting a region with a positive sign, it is possible to extract parts where the surface faces downward.

[0194] Figures 19(d) and (f) show the results of calculating approximate values ​​of the angle-dependent components when illuminating from a specific direction of β=45° (or 225°) for only each of the extracted regions, and adding them to the angle-independent components. In other words, Figures 19(c) to (e) are used as a mask for the angle-dependent components in Figure 19(b). This shows that it is possible to distinguish between illumination in the 45° and 225° directions depending on the up / down orientation of the surface, that is, it is possible to resolve the 180° ambiguity. EXAMPLES

[0195] Next, a fourth embodiment of the present invention will be described with reference to Figs. 20(a) to (f). Figs. 20(a) to (f) are diagrams showing the results of this embodiment. In this embodiment, six input images are obtained by capturing polarized light with polarization angles α of 0°, 45°, and 90° at two different viewpoints obtained by the same method as in the third embodiment. Then, an output image is generated that resolves the 180° ambiguity of the illumination for the object by utilizing the sign of the difference in the polarization degree between the viewpoints. The object is a kettle figure shown in Fig. 20(a) (these are binocular images when α=0°).

[0196] 20(b) shows the result of calculating A, φ, and Ic in equation (40) using both-eye images with polarization angles α of 0°, 45°, and 90°, and then synthesizing an image in which the object is illuminated from a specific direction β=0° with kc set to 1 and kd set to 1.5 in equation (42). Here, however, Id(α) in equation (42) is replaced with the function F(α) expressed by the following equation (45).

[0197]

number

[0198] The 180° period in the polarization angle makes it impossible to distinguish between 0° and 180° with respect to the x-axis, and it can be seen that both the left- and right-facing faces of the kettle body are imaged as being illuminated by a narrow beam of light.

[0199] Figures 20(c) and (e) show the results of region extraction using the sign of the difference in the degree of polarization calculated for the left-eye image and the right-eye image. In Figure 20(c), the difference sign is negative, and in Figure 20(e), the positive sign is shown in white. As mentioned above, it can be seen that by selecting a region with a negative sign, it is possible to extract parts where the surface faces left, and by selecting a region with a positive sign, it is possible to extract parts where the surface faces right.

[0200] Figures 20(d) and (f) show the results of calculating the approximate values ​​of the angle-dependent components when illuminating only each of the extracted regions from a specific direction of β=0° (or 180°) using the above F(α) and adding them to the angle-independent components. In other words, Figures 20(c) to (e) are used as a mask for the angle-dependent components in Figure 20(b). This shows that it is possible to distinguish between illumination in the 0° and 180° directions depending on the left-right orientation of the surface, that is, it is possible to resolve the 180° ambiguity. EXAMPLES

[0201] Next, a fifth embodiment of the present invention will be described with reference to Figs. 21(a) to (d). Figs. 21(a) to (d) are diagrams showing the results of this embodiment. In the first and second embodiments, an output image is generated by illuminating an object from one direction, but in this embodiment, an output image is generated by illuminating an object from multiple directions. The viewpoint in the input image, the polarization angle at which the image is captured, and the object are the same as in the second embodiment. Therefore, A, φ, and Ic in the calculated formula (40) are also the same as in the second embodiment. Furthermore, kc, kd, and Id(α) in the formula (42) are also the same as in the second embodiment. That is, Id(α) is replaced with the function F(α) in the formula (45).

[0202] In this embodiment, the multiple specific directions are β1=45° and β2=120°. This is because in everyday life, there are many cases where lighting is applied from above an object. For this reason, we want to extract the parts of the object whose surfaces face upward. Therefore, taking into consideration that the parallax in this embodiment is in the horizontal direction, we use the sign of the difference between the left-eye image and the right-eye image in the polarization direction. Figure 21(a) shows the result of region extraction, and it can be seen that the region whose surface faces upward has been extracted.

[0203] 21(b) and (c) show the results of calculating the approximate value Id' of the angle-dependent component in each specific direction of β1 and β2 in the extracted region (using F(α) in equation (45)) and adding it to the angle-independent component. From this, it can be seen that illumination is correctly achieved from the direction of 45° or 120° without causing 180° ambiguity.

[0204] Finally, the approximate values ​​Id'(45°) and Id'(120°) of the angle-dependent components for two specific directions, 45° and 120°, are summed up and added to the angle-independent components, as shown in Fig. 21(d). This makes it possible to naturally express a lighting condition in which spotlights are shining from two directions on an upward-facing surface. EXAMPLES

[0205] Next, a sixth embodiment of the present invention will be described with reference to Fig. 22. Fig. 22 is a block diagram of an image processing system 600 in this embodiment. The image processing system 600 includes an image processing device 601, an imaging device 602, a storage medium 603, a display device 604, and an output device 605.

[0206] The image processing device 601 is equipped with image processing software (image processing program as a computer program) 606. The image processing device 601 executes the image processing of steps ST1 to ST4 described above in accordance with the image processing software 606, thereby performing polarization analysis, area extraction, and image synthesis.

[0207] The imaging device 602 is a device that captures images, such as a camera, a telescope, an endoscope, etc. An example of a camera is a lens-interchangeable digital camera, which is a device that has an interchangeable lens, an adapter, a CMOS image sensor, etc. The storage medium 603 is a semiconductor memory, a hard disk, a server on a network, etc., that stores the images captured by imaging.

[0208] The image processing device 601 acquires an input image from the imaging device 602 or the storage medium 603 by wired or wireless communication with them or by attachment and reading. Then, an output image is generated by the image processing described above, and the output image is output to at least one of the output device 605, the imaging device 602, and the storage medium 603. The output image can also be saved in a storage unit (internal memory) built into the image processing device 601. The output device 605 is, for example, a printer.

[0209] A display device 604 is connected to the image processing device 601. Therefore, a user can perform image processing and evaluate the generated output image via the display device 604. In addition to the image processing described above, the image processing device 601 may also perform development processing and image restoration processing as necessary.

[0210] As described above, in the third to sixth embodiments, the image processing device 601 can execute the first process, the second process, and the third process. The first process acquires captured images captured at different polarization angles from a plurality of different viewpoints (a plurality of captured images obtained by capturing an object while changing the polarization angle of light from the object). The second process acquires polarization information of the object. The third process generates a composite image in which the illumination state of the object is changed for the captured image, thereby generating a composite image. The second process acquires light intensity information that changes according to the polarization angle using the captured image. The third process extracts a partial area of ​​the image area of ​​the captured image or the composite image using the sign of the difference in polarization information for a plurality of different viewpoints, and generates a composite image using the extracted partial area and the light intensity information.

[0211] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0212] According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, and a program capable of synthesizing images illuminated from any direction using polarization information.

[0213] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0214] 304 Image processing unit (image processing device)

Claims

1. a first process of acquiring a plurality of first images obtained by capturing images of an object in a first illumination state from a first and a second viewpoint while changing a polarization angle of light from the object; a second process of acquiring light intensity information that changes according to the polarization angle using the plurality of first images, and acquiring polarization information of the object based on the light intensity information; and a third operation of generating a second image of the object corresponding to a second lighting condition different from the first lighting condition. In the third processing, a partial area of ​​the first image is extracted using a sign of a difference between the polarization information at the first viewpoint and the polarization information at the second viewpoint, and the second image is generated by changing a luminance value in the partial area based on information on the second lighting condition and the light intensity information.

2. 2. The image processing device according to claim 1, wherein the polarization information is information regarding at least one of a polarization direction and a polarization degree of light from the subject.

3. 3. The image processing apparatus according to claim 2, wherein the sign is a sign of a difference between the polarization directions.

4. The image processing apparatus according to claim 2 , wherein the sign is a sign of the difference in the degree of polarization.

5. 5. An imaging apparatus comprising: the image processing device according to claim 1; and an imaging element for imaging the subject via an optical system.

6. 6. The imaging device according to claim 5, wherein the imaging element includes a first imaging unit that performs imaging at the first viewpoint by receiving light from a first region in a pupil of the optical system, and a second imaging unit that performs imaging at the second viewpoint by receiving light from a second region in the pupil.

7. 7. The imaging device according to claim 6, wherein the first and second imaging sections are disposed at different positions in a longitudinal direction of the imaging element.

Images