Image processing device, image processing method, and program

Abstract

An identical-region detection unit detects, from multiple-polarization-direction polarization images acquired at a plurality of different viewpoints, identical regions to be processed for the respective viewpoints. The multiple-polarization-direction polarized images acquired for each of the plurality of different viewpoints are acquired by imaging over multiple frame periods during which the positional relationship between an object to be processed and a polarization image acquisition unit that acquires the polarization images changes. A degree-of-polarization calculation unit calculates the degree of polarization the identical region for each viewpoint on the basis of the multiple-polarization-direction polarization images. A reflection removal unit performs a reflection removal process on the identical regions to be processed, using the multiple-polarization-direction polarization image from the viewpoint at which the degree of polarization calculated by the polarization calculation unit is greatest. Thus, the reflection component can be removed even if the angle formed by the planar direction of the reflection surface and the imaging direction is not clear.

Description

[Technical field]

[0001] This technique relates to an image processing device, an image processing method, and a program, and makes it possible to remove reflected components even if the angle between the surface direction of the reflecting surface and the shooting direction is not clear. [Background technology]

[0002] Conventionally, in addition to color images, polarized images are simultaneously acquired as described in Patent Document 1, and the degree of polarization and normals are calculated based on the polarized images, and the diffuse reflection component and specular reflection component are separated, etc. Patent Document 2 also describes that the reflection angle, which is the angle between the surface direction corresponding to the normal direction of the reflecting surface and the shooting direction, is input, so that the reflection component can be removed even if the reflection angle does not form the Brewster's angle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-290895 A [Patent Document 2] International Publication No. 2018 / 092540 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when using a polarized image to remove the reflected component of a color image, the method described in Patent Document 2 requires inputting the reflection angle, and therefore cannot handle reflection surfaces that have a different reflection angle from the input reflection angle.

[0005] Therefore, an object of this technology is to provide an image processing device, an image processing method, and a program that can remove reflected components even if the angle between the surface direction of the reflecting surface and the shooting direction is not clear. [Means for solving the problem]

[0006] The first aspect of this technology is: an identical region detection unit that detects identical regions to be processed for each of a plurality of viewpoints from polarized images having a plurality of polarization directions acquired for each of the plurality of different viewpoints; a polarization degree calculation unit that calculates a polarization degree of the same region for each of the viewpoints based on the polarized images in the plurality of polarization directions; a reflection removal unit that performs a reflection removal process on the same region of the processing target using the polarized images of the plurality of polarization directions at a viewpoint where the degree of polarization calculated by the polarization degree calculation unit is maximum; The present invention relates to an image processing device comprising:

[0007] In this technology, the same area of ​​the processing target is detected for each viewpoint by a same area detection unit from polarized images with multiple polarization directions acquired for each of multiple different viewpoints. The polarized images with multiple polarization directions acquired for each of multiple different viewpoints are polarized images acquired by capturing images over a multiple frame period in which the positional relationship between the processing target and the polarized image acquisition unit that acquires the polarized images changes, for example, a polarized image acquired by capturing an image of the processing target that moves, or a polarized image acquired by capturing an image with a moving polarized image acquisition unit. Also, the polarized images with multiple polarization directions acquired for each of multiple different viewpoints are polarized images acquired by providing a polarized image acquisition unit that acquires polarized images at multiple viewpoint positions and capturing an image of the processing target with each polarized image acquisition unit.

[0008] The polarization degree calculation unit calculates the polarization degree of the same region for each viewpoint on a pixel-by-pixel basis based on the polarized images with the multiple polarization directions, and the reflection removal unit performs reflection removal processing of the same region of the processing target using the polarized images with the multiple polarization directions of the viewpoint at which the polarization degree calculated by the polarization degree calculation unit is maximum. Also, when the position of the processing target is fixed, the position of the viewpoint at which the polarization degree calculated by the polarization degree calculation unit is maximum may be stored together with the position of the processing target, and the reflection removal unit may perform reflection removal processing using the polarized image of the processing target acquired by the polarized image acquisition unit at the viewpoint position at which the polarization degree is maximum.

[0009] The second aspect of this technology is detecting an identical region to be processed from a plurality of polarization directions of polarized images acquired for each of a plurality of different viewpoints by an identical region detection unit; calculating a degree of polarization of the same region for each of the viewpoints based on the polarized images of the plurality of polarization directions by a polarization degree calculation unit; a reflection removing unit performing a reflection removing process on the same region of the processing target using the polarized images of the plurality of polarization directions at a viewpoint where the degree of polarization calculated by the polarization degree calculating unit is maximum; The present invention relates to an image processing method including the steps of:

[0010] The third aspect of this technology is A program for causing a computer to execute removal of a reflected component to be processed, detecting the same area to be processed for each of a plurality of viewpoints from polarization images having a plurality of polarization directions acquired for each of the plurality of different viewpoints; calculating a degree of polarization of the same region for each of the viewpoints based on the polarized images of the plurality of polarization directions; performing a reflection removal process for the same region of the processing target using the polarized images of the plurality of polarization directions at the viewpoint where the calculated polarization degree is maximum; The program is executed by the computer.

[0011] The program of the present technology is, for example, a program that can be provided in a computer-readable format to a general-purpose computer capable of executing various program codes, via a storage medium, a communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By providing such a program in a computer-readable format, processing according to the program is realized on the computer. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram for explaining P waves and S waves. [Diagram 2] This is a diagram showing the specular reflection of S and P waves versus the angle of incidence. [Diagram 3]11A and 11B are diagrams illustrating changes in luminance of a polarized image. [Figure 4] FIG. 13 is a diagram illustrating the relationship between luminance and polarization angle. [Diagram 5] FIG. 1 is a diagram illustrating a configuration of an embodiment. [Figure 6] 4 is a diagram illustrating an example of the configuration of a polarization image acquisition unit. FIG. [Figure 7] 1 is a diagram illustrating pixel configurations for a plurality of polarization directions. [Figure 8] 4 is a flowchart illustrating an operation of the embodiment. [Figure 9] FIG. 13 is a diagram showing an operation example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Principles and issues of reflected light removal processing 2. Configuration of the embodiment 3. Operation of the embodiment 4. Application Examples

[0014] <1. Principles and issues of reflected light removal processing> Next, the principles and issues of reflected light removal processing will be explained. Figure 1 is a diagram for explaining P waves and S waves. P waves are light that has only components parallel to the incident surface, and S waves are light that has only components perpendicular to the incident surface. Figure 2 shows the specular reflection of S waves and P waves against the angle of incidence. Note that the solid line shows the characteristics of S waves, and the dashed line shows the characteristics of P waves.

[0015] It is known that the reflectance of P waves is nearly zero when the angle of incidence (= reflection angle) is around 70 degrees, and the angle at which the reflectance of P waves is approximately "0" is called the Brewster angle. Therefore, when the angle of incidence is the Brewster angle, the reflected light does not contain any P wave components, but only S wave components, so the reflected components can be removed by removing the S wave components with a polarizing filter. However, when the angle of reflection is away from the Brewster angle, the reflected light contains both S wave components and P wave components, so it is not possible to obtain an image in which the reflected components have been removed by reducing the S wave components with a polarizing filter.

[0016] Therefore, the image processing device of the present disclosure solves the problem of the angle between the surface direction of the reflecting surface and the shooting direction, and performs processing to remove or reduce reflected light components not only from images shot in a specific direction limited by the Brewster angle but also from images shot from various directions to obtain a clear subject image (transmitted light image or transmitted light component image). Specifically, the same area to be processed is detected from polarized images with multiple polarization directions acquired for multiple different viewpoints, the polarization degree of the same area is calculated for each viewpoint, and reflection removal processing of the same area is performed using polarized images with multiple polarization directions from the viewpoint where the polarization degree is maximum.

[0017] FIG. 3 is a diagram for explaining the change in luminance of a polarized image. As shown in FIG. 3, the object OB is illuminated using a light source LT, and the object OB is imaged by the imaging unit CM through the polarizing plate PL. In this case, it is known that the luminance of the object OB changes in the polarized image generated by the imaging unit CM according to the rotation of the polarizing plate PL. Here, the highest luminance when the polarizing plate PL is rotated is Imax, and the lowest luminance is Imin. In addition, when the x-axis and y-axis in the two-dimensional coordinate are the planar directions of the polarizing plate PL, the angle on the xy plane with respect to the x-axis when the polarizing plate PL is rotated is the polarization angle υpol. When the polarizing plate PL is rotated 180 degrees, it returns to its original polarization state and has a period of 180 degrees.

[0018] Figure 4 illustrates the relationship between brightness and polarization angle. The polarization angle υpol at which the maximum brightness Imax is observed is defined as the azimuth angle φ. With this definition, the polarization model equation showing the change in brightness Ipol observed when the polarizing plate PL is rotated, that is, the specified brightness change caused by the difference in polarization angle, can be expressed as equation (1).

[0019]

number

[0020] In equation (1), the polarization angle υpol is clear when the polarized image is generated, and the maximum brightness Imax, minimum brightness Imin, and azimuth angle φ are variables. Therefore, since there are three variables, it is possible to fit to the function shown in equation (1) by using, for example, the brightness of a polarized image with three or more polarization directions. In addition, the degree of polarization ρ can be calculated by calculating equation (2) using the maximum brightness Imax and minimum brightness Imin. Also, as is clear from equation (2), the degree of polarization ρ increases when the specular reflection component is large.

[0021]

number

[0022] In addition, specular reflection occurs due to a dominant light source in the imaging scene. In addition, since the white balance correction adjusts the correction value according to the color of the illumination, it is considered that the color of the illumination that generates the specular reflection is achromatic. In this case, the RGB values ​​representing the color of the illumination are the same value, and it is possible to remove the specular reflection component by calculating the color difference. Therefore, in removing the specular reflection, as shown in the patent document "International Publication No. 2016-136085", for example, a calculation is performed using the pixel values ​​of the red pixels, the pixel values ​​of the green image, and the pixel values ​​of the blue pixels in the same polarization pixel block from a polarized image that has been subjected to white balance correction or the like, to generate a polarized image from which the specular reflection component has been removed. In addition, in removing the specular reflection, for example, a polarized image from which the specular reflection component due to the light source has been removed may be generated under the assumption that the light source is white.

[0023] 2. Configuration of the embodiment Next, the configuration of the embodiment will be described. An example of the configuration of the embodiment is shown in Fig. 5. An image processing system 10 includes a polarized image acquisition section 20 and an image processing section 30.

[0024] The polarized image acquisition section 20 captures a polarized image by capturing an image of a subject. FIG. 6 illustrates an example of the configuration of the polarized image acquisition section. The polarized image acquisition section 20 generates an image by capturing an image with a polarizing plate 202 having a pixel configuration with a plurality of polarization directions arranged on an image sensor 201, as shown in FIG. 6(a), for example. Note that FIG. 6(a) illustrates a case where a polarizing plate 202, in which each pixel has one of four different polarization directions (the polarization directions are indicated by arrows), is arranged in front of the image sensor 201. Also, the polarized image acquisition section 20 may generate a plurality of polarized images with different polarization directions by utilizing a multi-lens array configuration, as shown in FIG. 6(b). For example, a plurality of lenses 203 (four in the figure) are provided in front of the image sensor 201, and the optical images of the subject are formed on the imaging surface of the image sensor 201 by the respective lenses 203. Furthermore, a polarizing plate 204 is provided in front of each lens 203, and the polarization direction of the polarizing plate 204 (the polarization direction is indicated by an arrow) is set to a different direction to generate a polarized image made up of pixels with a plurality of polarization directions. By configuring the polarized image acquisition section 20 in this manner, a polarized image made up of pixels with a plurality of polarization directions can be acquired in one imaging session. Furthermore, as shown in (c) of FIG. 6, a configuration may be used in which polarizing plates 212-1 to 212-4 with different polarization directions are provided in front of the imaging sections 210-1 to 210-4 to generate a plurality of polarized images with different polarization directions.

[0025] In addition, when the subject moves slowly or moves in steps, a polarizing plate 211 may be provided in front of the imaging unit 210 as shown in Fig. 6(d). In this case, the polarizing plate 211 is rotated to capture images in a plurality of different polarization directions, thereby acquiring a plurality of polarized images with different polarization directions.

[0026] The different polarization directions may be any combination of angles as long as the angles are all different. For example, when three polarization directions are used, 0 degrees, 60 degrees, and 120 degrees are used, and when four polarization directions are used, 0 degrees, 45 degrees, 90 degrees, and 135 degrees are used.

[0027] When the image sensor 201 does not use a color filter, the polarized image acquisition unit 20 can acquire a luminance polarized image. Here, in the case of (a) in FIG. 6, an image equivalent to a normal luminance image of non-polarized light can be acquired by averaging the luminance of four adjacent pixels with different polarization directions. Also, in the cases of (b) and (c) in FIG. 6, if the position intervals of the lenses 203 and the imaging units 210-1 to 210-4 are short enough to be negligible with respect to the distance to the abnormality detection target, the parallax can be ignored in a plurality of polarized images with different polarization directions. Therefore, an image equivalent to a normal luminance image of non-polarized light can be acquired by averaging the luminance of the polarized images with different polarization directions. Also, if the parallax cannot be ignored, an image equivalent to a normal luminance image of non-polarized light can be acquired by aligning the polarized images with different polarization directions according to the amount of parallax and averaging the luminance of the polarized images after the alignment. Also, in the case of (d) in FIG. 6, an image equivalent to a normal luminance image of non-polarized light can be acquired by averaging the luminance of the luminance polarized images with different polarization directions for each pixel.

[0028] Furthermore, the polarization image acquisition section 20 may generate not only a luminance polarization image, but also a three-primary color image by providing a color filter in the image sensor 201, and may simultaneously generate an infrared image, etc. The polarization image acquisition section 20 may also generate a luminance image by calculating the luminance from the three-primary color image.

[0029] FIG. 7 illustrates pixel configurations for a plurality of polarization directions, and the configuration shown in FIG. 7 is repeated in the horizontal and vertical directions. (a) and (b) of FIG. 7 illustrate pixel configurations when acquiring a black-and-white image. (a) of FIG. 7 illustrates a case where a 2×2 pixel polarization pixel block is configured with polarization pixels having polarization directions (polarization angles) of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees. (b) of FIG. 7 illustrates a case where a 4×4 pixel polarization pixel block is configured with polarization pixels having polarization directions of, for example, 0 degrees, 45 degrees, 90 degrees, and 135 degrees, with 2×2 pixels as the polarization direction unit. When the polarization component unit of the polarizing plate is 2×2 pixels as shown in (b) of FIG. 7, the ratio of the leakage of polarization components from the area of ​​the adjacent different polarization component unit to the polarization components obtained for each polarization component unit is smaller than that of the 1×1 pixel shown in (a) of FIG. 7. Furthermore, when a wire grid is used as the polarizing plate, polarized light whose electric field component is perpendicular to the lattice direction (wire direction) is transmitted, and the longer the wire, the higher the transmittance. Therefore, when the polarization component unit is 2x2 pixels, the transmittance is higher than that of a 1x1 pixel. Therefore, when the polarization component unit is 2x2 pixels, the transmittance is higher than that of a 1x1 pixel, and the extinction ratio can be improved.

[0030] (c) to (g) of Fig. 7 show examples of pixel configurations when acquiring a color image. (c) of Fig. 7 shows a case where the 2 × 2 pixel polarization pixel block shown in (a) of Fig. 7 is treated as one color unit, and the three primary color pixels (red pixel, green pixel, and red pixel) are arranged in a Bayer array.

[0031] FIG. 7D illustrates an example in which three primary color pixels are arranged in a Bayer array for each pixel block of 2×2 pixels having the same polarization direction shown in FIG. 7B.

[0032] FIG. 7(e) illustrates a case in which three primary color pixels are arranged in a Bayer array for each 2×2 pixel block with the same polarization direction, and 2×2 pixel blocks with different polarization directions are pixels of the same color.

[0033] (f) of Figure 7 shows a case where a pixel block of 2 x 2 pixels in a Bayer array has the same polarization direction, and the phase difference in the polarization direction between horizontally adjacent pixel blocks is 90 degrees, and the phase difference in the polarization direction between vertically adjacent pixel blocks is ±45 degrees.

[0034] (g) of Figure 7 shows a case where a pixel block of 2 x 2 pixels in a Bayer array has the same polarization direction, and the phase difference in the polarization direction between vertically adjacent pixel blocks is 90 degrees, and the phase difference in the polarization direction between horizontally adjacent pixel blocks is ±45 degrees.

[0035] In addition, in the pixel configuration, a 2×2 pixel block may be configured with three primary color pixels and white pixels, or may be configured with a three primary color pixel block and a white pixel block. Furthermore, a 2×2 pixel block may be configured with polarized pixels and non-polarized pixels with different polarization directions, or may be configured with a polarized pixel block and a non-polarized pixel block with different polarization directions. When such a pixel configuration is used, the polarization image acquisition unit 20 generates a polarization imaging pixel with a resolution in pixel units, that is, in pixel block units of a predetermined number of pixels, for each polarization direction. Also, when the pixel block is in a Bayer array, a polarization imaging pixel with a resolution in one pixel unit may be generated for each polarization direction using an existing demosaic process.

[0036] The polarized image acquisition section 20 acquires polarized images with multiple polarization directions for each of multiple different viewpoints. For example, the polarized image acquisition section 20 may acquire polarized images with multiple polarization directions for each of multiple different viewpoints by capturing an image of a moving subject over a period of multiple frames as shown in FIG. 2(a), FIG. 2(b), or FIG. 2(c). Alternatively, the polarized image acquisition section 20 may be provided at multiple different viewpoints and capture an image of the subject, thereby acquiring polarized images with multiple polarization directions by each of the multiple polarized image acquisition sections 20.

[0037] Returning to FIG. 5, the image processing unit 30 includes an identical region detection unit 31, a polarization degree calculation unit 32, a storage unit 33, an image selection unit , and a reflection removal unit .

[0038] The identical area detection unit 31 detects identical areas to be processed for each viewpoint from the polarized images acquired by the polarized image acquisition unit 20. For example, the identical area detection unit 31 detects identical areas to be processed using the polarized images acquired by the polarized image acquisition unit 20 and polarized images stored in the storage unit 33 described below. The identical area detection unit 31 outputs the identical area detection result to the image selection unit 34. When detecting identical areas to be processed from each polarized image, if the identical area detection unit 31 corrects changes in shape in the polarized images of the subject caused by the movement of the subject or differences in the viewpoint position of the polarized image acquisition unit 20, the detection of identical areas becomes easier by using the corrected polarized images.

[0039] The polarization degree calculation unit 32 calculates the polarization degree for each pixel of the polarized image for each of the different viewpoints. The polarization degree calculation unit 32 calculates the polarization degree for each pixel based on formula (2) using the polarized images with the different polarization directions. The polarization degree calculation unit 32 also calculates the polarization degree for each of the different viewpoints and outputs the calculated polarization degree to the storage unit 33.

[0040] The storage unit 33 stores the polarized images with the multiple polarization directions acquired for each of the multiple different viewpoints by the polarized image acquisition unit 20. The storage unit 33 also stores the degree of polarization calculated on a pixel-by-pixel basis for each of the multiple viewpoints.

[0041] The image selection unit 34 reads out the degrees of polarization of the identical region detected by the identical region detection unit 31 from the storage unit 33 for each of the multiple viewpoints. The image selection unit 34 also compares the read degrees of polarization to determine the viewpoint with the maximum degree of polarization. For example, when the identical region is a multiple pixel region, the image selection unit 34 determines the viewpoint with the maximum degree of polarization for each pixel position. The image selection unit 34 also reads out the polarized images for each of the multiple polarization directions acquired at the viewpoint with the maximum degree of polarization from the storage unit 33, and outputs them to the reflection removal unit 35.

[0042] The reflection removal unit 35 removes the specular reflection component by using the polarized image selected by the image selection unit 34. Note that the specular reflection component may be removed by using a method described in, for example, Patent Document "International Publication No. WO 2016 / 136085" or Patent Document "International Publication No. WO 2018 / 230119".

[0043] 3. Operation of the embodiment 8 is a flow chart illustrating the operation of the embodiment. In step ST1, the image processing unit acquires a polarized image. The image processing unit 30 acquires polarized images in multiple polarization directions for multiple different viewpoints from the polarized image acquisition unit 20, and proceeds to step ST2.

[0044] In step ST2, the image processing unit detects identical regions. Identical region detection unit 31 of image processing unit 30 detects identical regions to be processed for each viewpoint from the polarized image acquired in step ST1, and the process proceeds to step ST3.

[0045] In step ST3, the image processing unit calculates the degree of polarization. The polarization degree calculation unit 32 of the image processing unit 30 calculates the degree of polarization for each pixel and for each viewpoint using the polarized images with the multiple polarization directions, and then the process proceeds to step ST4.

[0046] In step ST4, the image processing unit selects a polarized image. The image selection unit 34 of the image processing unit 30 determines the viewpoint at which the degree of polarization calculated in step ST3 is maximized for each pixel position in the same region detected in step ST2, selects the polarized image at the viewpoint at which the degree of polarization is maximized, and proceeds to step ST5.

[0047] In step ST5, the image processing unit performs a reflection removal process. The reflection removal unit 35 of the image processing unit 30 removes the specular reflection component using the polarized image selected in step ST4, and converts the image of the same area detected in step ST2 into an image in which reflection has been removed or suppressed.

[0048] 9 shows an example of the operation of the embodiment, in which the processing target is a moving vehicle OBm, the polarized image acquisition unit 20 is fixedly provided at the front upper part of the vehicle OBm, and captures the vehicle OBm to acquire a polarized image. The normal of the area FP on the windshield of the vehicle OBm is the normal SNfp. The image processing unit 30 (not shown) uses the polarized images acquired by capturing images over a period of multiple frames in which the positional relationship between the processing target, for example, the vehicle OBm and the polarized image acquisition unit 20 that acquires the polarized image changes, as polarized images of multiple polarization directions acquired for each of multiple different viewpoints, to perform a reflection removal process on the area FP on the windshield of the vehicle OBm.

[0049] FIG. 9(a) illustrates an example in which the vehicle OBm is at position P1 at time t1. In this case, the light emitted from the region FP enters the polarized image acquisition unit 20 at a reflection angle of α1. FIG. 9(b) illustrates an example in which the vehicle OBm is at position P2 at time t2. In this case, the light emitted from the region FP enters the polarized image acquisition unit 20 at a reflection angle of α2. FIG. 9(c) illustrates an example in which the vehicle OBm is at position P1 at time t3. In this case, the light emitted from the region FP enters the polarized image acquisition unit 20 at a reflection angle of α3.

[0050] The image processing unit 30 calculates the degree of polarization based on the polarized images of the multiple polarization directions in the region FP from the polarized images acquired by the polarized image acquisition unit 20 at each time point. Furthermore, the image processing unit 30 selects the polarized image with the maximum degree of polarization. For example, when the reflection angle α2 is the Brewster angle, the degree of polarization is maximum, so the image processing unit 30 uses the polarized images of the multiple polarization directions showing the region FP acquired at time t2 at which the degree of polarization is maximum. In this case, since there are few P waves in the light emitted from the region FP at time t2, it is possible to obtain a captured image with reduced specular reflection by performing the reflection removal process using the polarized images of the multiple polarization directions showing the region FP acquired at time t2, even without setting the angle of the emitted light, etc.

[0051] In addition, although FIG. 9 illustrates an example in which the processing target moves, the position of the processing target may be fixed and the polarized image acquisition unit may move. In addition, when the position of the processing target is fixed and the polarized image acquisition unit moves, the position of the viewpoint at which the polarization degree calculated by the polarization degree calculation unit is maximized may be stored together with the position of the processing target. In this way, if the position of the processing target and the position of the viewpoint at which the polarization degree is maximized are stored, when the polarized image acquisition unit is provided in each moving body, the reflection component can be easily removed by using the polarized image acquired at the position based on the stored information without having to calculate the polarization degree in each moving body or determine the position of the viewpoint at which the calculated polarization degree is maximized. For example, when a building or the like that generates reflection is indicated by map information, the polarized image can be acquired at the position of the viewpoint that is stored in association with the position of the building that generates reflection, and the reflection generated by the building or the like can be easily removed.

[0052] <4. Application Examples> The technology according to the present disclosure can be applied to various fields. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility, an airplane, a drone, a ship, a robot, etc. Also, the technology may be realized as a device mounted on equipment used in a production process in a factory or equipment used in the construction field. When the image processing device of the present technology is applied to such fields, it is possible to easily remove reflected components, so that a system that enables safer driving, etc. can be constructed. Also, when applied to the surveillance field, etc., it is possible to easily remove reflected components of a moving target to be monitored.

[0053] The series of processes described in this specification can be executed by hardware, software, or a combination of both. When executing processes by software, a program recording the processing sequence is installed in the memory of a computer built into dedicated hardware and executed. Alternatively, the program can be installed in a general-purpose computer capable of executing various processes and executed.

[0054] For example, the program can be pre-recorded on a recording medium such as a hard disk, a solid state drive (SSD), or a read only memory (ROM). Alternatively, the program can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a compact disc read only memory (CD-ROM), a magneto optical (MO) disk, a digital versatile disc (DVD), a Blu-Ray Disc (BD (registered trademark)), a magnetic disk, or a semiconductor memory card. Such removable recording media can be provided as so-called package software.

[0055] In addition to being installed in a computer from a removable recording medium, the program may be transferred wirelessly or by wire from a download site to a computer via a network such as a LAN (Local Area Network) or the Internet. The computer can receive the program transferred in this manner and install it in a built-in recording medium such as a hard disk.

[0056] In addition, the effects described in this specification are merely examples and are not limited, and there may be additional effects that are not described. In addition, the present technology should not be interpreted as being limited to the above-mentioned embodiments of the technology. The embodiments of the technology disclose the present technology in the form of examples, and it is self-evident that a person skilled in the art can modify or substitute the embodiments without departing from the gist of the present technology. In other words, the scope of the claims should be taken into consideration in order to determine the gist of the present technology.

[0057] Moreover, the image processing device of the present technology can also have the following configuration. (1) an identical area detection unit that detects identical areas to be processed for each of a plurality of viewpoints from polarized images having a plurality of polarization directions acquired for each of the plurality of different viewpoints; a polarization degree calculation unit that calculates a polarization degree of the same region for each of the viewpoints based on the polarized images in the plurality of polarization directions; a reflection removal unit that performs a reflection removal process on the same region of the processing target using the polarized images of the plurality of polarization directions at a viewpoint where the degree of polarization calculated by the polarization degree calculation unit is maximum; An image processing device comprising: (2) The image processing device described in (1), wherein the polarized images of multiple polarization directions acquired for each of the different multiple viewpoints are polarized images acquired by capturing images over a multiple frame period during which the positional relationship between the object to be processed and a polarized image acquisition unit that acquires the polarized images changes. (3) The image processing device according to (2), wherein the polarized image is a polarized image acquired by capturing an image of the processing target moving. (4) The image processing device according to (2), wherein the polarized image is a polarized image acquired by capturing an image with the moving polarized image acquisition unit. (5) The image processing device described in (1), wherein the polarized images of the multiple polarization directions acquired for each of the different multiple viewpoints are polarized images acquired by providing a polarized image acquisition unit for acquiring the polarized images at multiple viewpoint positions and capturing an image of the processing target with each polarized image acquisition unit. (6) The image processing device according to any one of (1) to (5), wherein the degree of polarization calculation unit calculates the degree of polarization on a pixel-by-pixel basis. (7) An image processing device according to any one of (1) to (6), which, when the position of the processing object is fixed, stores the viewpoint position at which the degree of polarization calculated by the polarization degree calculation unit is maximum together with the position of the processing object. (8) The image processing device according to (7), wherein the reflection removal unit performs the reflection removal process using a polarized image of the processing target acquired by a polarized image acquisition unit at a viewpoint position where the degree of polarization is maximum. [Explanation of symbols]

[0058] 10. Image Processing System 20....Polarized image acquisition section 30...Image processing unit 31....Identical region detection unit 32 Polarization degree calculation section 33...Storage section 34 Image selection section 35...Reflection removal section 201...Image sensor 202, 204, 211, 212-1 to 212-4... Polarizing plate 203···Lens 210, 210-1 to 210-4: Imaging unit

Claims

1. an identical region detection unit that detects identical regions to be processed for each of a plurality of viewpoints from polarized images having a plurality of polarization directions acquired for each of the plurality of different viewpoints; a polarization degree calculation unit that calculates a polarization degree of the same region for each of the different viewpoints based on the polarized images having a plurality of polarization directions acquired for each of the different viewpoints; a reflection removal unit that performs a reflection removal process on the same region of the processing target using the polarized images of the plurality of polarization directions at a viewpoint where the degree of polarization calculated by the polarization degree calculation unit is maximum; An image processing device comprising:

2. The polarized images in the multiple polarization directions acquired for each of the multiple different viewpoints are polarized images acquired by capturing images over a multiple frame period during which the positional relationship between the processing object and a polarized image acquisition unit that acquires the polarized images changes. The image processing device according to claim 1 .

3. The polarized image is a polarized image acquired by capturing an image of the processing target that is moving. The image processing device according to claim 2 .

4. The polarized image is a polarized image acquired by capturing an image with the moving polarized image acquisition unit. The image processing device according to claim 2 .

5. The polarized images in the multiple polarization directions acquired for each of the multiple different viewpoints are polarized images acquired by providing polarized image acquisition units for acquiring the polarized images at multiple viewpoint positions and capturing images of the processing target with each polarized image acquisition unit. The image processing device according to claim 1 .

6. The polarization degree calculation unit calculates the polarization degree on a pixel-by-pixel basis. The image processing device according to claim 1 .

7. When the position of the processing object is fixed, the position of the viewpoint at which the degree of polarization calculated by the polarization degree calculation unit is maximized is stored together with the position of the processing object. The image processing device according to claim 1 .

8. The reflection removing unit performs the reflection removing process using the polarized image of the processing target acquired by a polarized image acquiring unit at a viewpoint position where the degree of polarization is maximized. The image processing device according to claim 7.

9. detecting an identical region to be processed from a plurality of polarization directions of polarized images acquired for each of a plurality of different viewpoints by an identical region detection unit; calculating a degree of polarization of the same region for each of the different viewpoints based on the polarized images having a plurality of polarization directions acquired for each of the different viewpoints by a polarization degree calculation unit; a reflection removing unit performing a reflection removing process on the same region of the processing target using the polarized images of the plurality of polarization directions at a viewpoint where the degree of polarization calculated by the polarization degree calculating unit is maximum; An image processing method comprising:

10. A program for causing a computer to execute removal of a reflected component to be processed, detecting the same area to be processed for each of a plurality of viewpoints from polarization images having a plurality of polarization directions acquired for each of the plurality of different viewpoints; calculating a degree of polarization of the same region for each of the different viewpoints based on polarized images having a plurality of polarization directions acquired for each of the different viewpoints; performing a reflection removal process for the same region of the processing target using the polarized images of the plurality of polarization directions at the viewpoint where the calculated polarization degree is maximum; A program for causing the computer to execute the above.

Images