Image processing device and method, and imaging device

Abstract

To perform polarization removal processing with higher accuracy by using an image signal having a plurality of different polarization properties obtained through single photographing.SOLUTION: Input of image signals generated by photoelectrically converting a plurality of rays of polarized light different in polarization angle from each other is received, and with a plurality of image signals based on the plurality of rays of polarized light as one set, for each set, the maximum value of the polarization intensities of polarization components of the plurality of image signals and a polarization angle at which the polarization intensity becomes maximum are determined. A target area for which polarization removal is performed is determined based on the image signals; a statistical area included in the target area for collecting statistics is set; and a representative intensity being the mode of the polarization intensities of the sets included in the statistical area and a representative angle being the mode of the polarization angles are obtained. The image signals in the sets included in the target area are corrected according to the polarization angles of the image signals by using the determined representative intensity and representative angle.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device and method, and an imaging device, and more particularly to a technique for processing image signals generated by photoelectrically converting a plurality of polarized lights having different polarization angles. [Background technology]

[0002] Conventionally, imaging devices, such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) sensors, have multiple light-receiving elements (pixels). Each pixel converts light into electricity, allowing it to detect the intensity (brightness) of that light. Furthermore, by placing a color filter in each pixel that primarily transmits light in one of the wavelength bands of red (R), green (G), or blue (B), it is possible to detect only the intensity of light of wavelengths (colors) of visible light. Using this mechanism, it is possible to record a subject in visible light as a video signal (electrical signal) in a storage device or display it on a display device.

[0003] In addition to elements such as brightness and color, light also has a property called polarization. Polarization can be thought of as the vibration direction of light, and it is known that light emitted from a light source has various vibration direction components (polarization directions) when reflected from a subject. However, in reality, a mixture of polarized and unpolarized light reaches the eye, so it is not possible to distinguish between polarized light components (polarization components).

[0004] On the other hand, it is generally known that the properties of polarization can be used to highlight desired reflected light or remove unwanted reflected light. For example, using a polarized light (PL) filter to remove images that are reflected on the surface of water or glass is a commonly used photography technique. Various applications are expected in the future as a way to utilize the unconventional properties of light, such as creating a contrast effect by suppressing unwanted reflected light or visualizing the stress on an object from the intensity of polarization.

[0005] Patent Document 1 proposes a technique for generating an output image corresponding to polarized light at a desired polarization angle using the brightness values ​​of multiple polarization images obtained by photoelectrically converting light at different polarization angles. This technique calculates an approximation function of the brightness component (polarization component) that changes depending on the polarization angle, calculates the polarization component at a specific polarization angle based on the calculated approximation function, and generates an output image corresponding to polarized light at the desired polarization angle. Using this technique, the polarization angle of the output image can be adjusted as desired.

[0006] Furthermore, Patent Document 1 shows an example of a method for acquiring multiple polarized images in which a polarizer array, in which four polarizers with different polarization directions are arranged as a set, is integrally provided to an imaging element, and four images with different polarization angles are generated from the output image signal. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2017-228910 Summary of the Invention [Problem to be solved by the invention]

[0008] However, when using the technology of Patent Document 1 to acquire images with different polarization angles using the polarizer array described above and remove polarization components at any polarization angle, the following problem occurs: The minimum luminance value of the approximation function is applied to the image signals of four pixels corresponding to each pair of polarizers used to obtain the approximation function, resulting in a decrease in resolution after polarization removal processing.

[0009] For example, when photographing a subject inside a car as shown in Fig. 35(a), consider a situation where, at a certain polarization angle, the windshield reflects light and the interior of the car cannot be photographed, as shown in area 3501 in Fig. 35(b). In this case, it is possible to photograph using an image sensor equipped with a polarizer array and remove the reflected light by performing polarization removal processing on the acquired image, but this results in a decrease in resolution.

[0010] Furthermore, if the four-pixel image signal used to calculate the approximation function contains edge components, the approximation function cannot be calculated correctly in the first place, and therefore the polarization component cannot be removed from the image signal containing the edge components.

[0011] The present invention has been made in consideration of the above problems, and aims to enable more accurate polarization removal processing using image signals having multiple different polarization characteristics obtained from a single capture. [Means for solving the problem]

[0012] In order to achieve the above-mentioned object, the image processing device of the present invention has an input means for inputting image signals generated by photoelectrically converting multiple polarized light beams having different polarization angles; a first processing means for treating multiple image signals based on the multiple polarized light beams as a set and for each set calculating the maximum polarization intensity of the polarization components of the multiple image signals and the polarization angle at which the polarization intensity is maximum; a setting means for determining a target area to be subjected to polarization removal based on the image signals and setting a statistical area included in the target area for taking statistics; a second processing means for determining a representative intensity which is the most frequent value of the polarization intensity of the set included in the statistical area and a representative angle which is the most frequent value of the polarization angle; and a correction means for performing a first correction process using the representative intensity and the representative angle to correct each image signal of the set included in the target area according to the polarization angle of each image signal. [Effects of the Invention]

[0013] According to the present invention, it is possible to perform polarization removal processing with higher accuracy by using image signals having a plurality of different polarization characteristics obtained by one image capture. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram showing the functional configuration of an imaging device according to first to fourth, sixth, and eighth to tenth embodiments of the present invention. [Figure 2] 4A and 4B are diagrams showing examples of arrangement of polarizing filters in the embodiment. [Figure 3] 3A and 3B are diagrams showing examples of the arrangement of color filters and polarizing filters in the embodiment. [Figure 4] 3A to 3C are diagrams showing examples of subjects in the first embodiment. [Figure 5] 5A and 5B are diagrams showing examples of images generated for different polarization directions when the subject shown in FIG. 4 is photographed. [Figure 6] 4 is a graph showing an approximation function of polarization components according to the first embodiment. [Figure 7] 4 is a flowchart showing a method for removing a polarized light component according to the first embodiment. [Figure 8] 4A to 4C are diagrams showing examples of histograms showing the occurrence frequencies of polarization intensities and polarization angles according to the first to fourth embodiments. [Figure 9] 10A to 10C are diagrams showing examples of subjects in second to fourth embodiments. [Figure 10] 10 is a flowchart showing a method for removing a polarized light component according to a second embodiment. [Figure 11] 10 is a flowchart showing a method for removing a polarized light component according to a third embodiment. [Figure 12] 13A to 13C are diagrams showing examples of exposure times and polarizing filter arrangements for HDR synthesis according to the fifth embodiment. [Figure 13] FIG. 1 is a diagram illustrating HDR synthesis. [Figure 14] A diagram explaining the issues that arise in HDR compositing when using a polarizing filter. [Figure 15] FIG. 10 is a block diagram showing the functional configuration of an imaging apparatus according to a fifth embodiment. [Figure 16] FIG. 13 is a diagram showing gain conversion for obtaining an approximate function of polarization components according to the fifth embodiment. [Figure 17] 13 is a flowchart of HDR synthesis processing using a polarization-removed image according to the fifth embodiment. [Figure 18] 10 is a flowchart showing a method for removing a polarized light component according to a sixth embodiment. [Figure 19] 13A to 13C are diagrams for explaining a local four-pixel shift method according to the sixth embodiment. [Figure 20] 13 is a flowchart showing a method for removing a polarized light component according to a modification of the sixth embodiment. [Figure 21] 13A and 13B are diagrams for explaining a method of shifting four local pixels in a modification of the sixth embodiment. [Figure 22] FIG. 13 is a block diagram showing the functional configuration of an imaging apparatus according to a seventh embodiment. [Figure 23] FIG. 13 is a diagram for explaining an interpolation method according to the seventh embodiment. [Figure 24] FIG. 13 is a diagram for explaining an interpolation method when a color filter according to the seventh embodiment is arranged. [Figure 25]FIG. 13 is a diagram for explaining a method for determining whether or not an edge exists according to the seventh embodiment. [Figure 26] 13 is a flowchart showing the flow of edge determination processing according to the seventh embodiment. [Figure 27] 13A and 13B are diagrams showing examples of subjects having multiple reflecting surfaces according to the eighth embodiment. [Figure 28] FIG. 20 is a diagram showing a histogram of polarized light components when the subject has multiple reflecting surfaces according to the eighth embodiment. [Figure 29] 13 is a flowchart showing a method for removing a polarized light component according to the eighth embodiment. [Figure 30] 13A to 13C are diagrams showing examples of subjects having curved surfaces according to the ninth embodiment. [Figure 31] FIG. 23 is a diagram showing a histogram of polarized light components when the subject has a curved surface according to the ninth embodiment. [Figure 32] 13 is a flowchart showing a method for removing a polarized light component according to a ninth embodiment. [Figure 33] 20 is a flowchart showing a method for removing a polarized light component according to a tenth embodiment. [Figure 34] FIG. 23 is a diagram for explaining an edge determination method according to the tenth embodiment. [Figure 35] FIG. 1 is a diagram for explaining a problem. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0016] First Embodiment A first embodiment of the present invention will be described below. 1 is a block diagram showing the functional configuration of an image capture device 100 according to a first embodiment of the present invention, to which an image processing device is applied, and shows functions related to the removal (correction) of polarization components. The image capture device 100 is an example of a device that can input an image, remove any polarization components from the input image, and output the image.

[0017] The image sensor 101 has a photoelectric conversion unit that converts incident light into an analog electrical signal, and an AD conversion unit that converts the converted analog signal into a digital signal. In addition, multiple polarizing filters with different polarization directions are arranged on the imaging surface of the image sensor 101 for each pixel.

[0018] The arrangement of polarizing filters in the pixel section 112 of the image sensor 101 used in this embodiment will now be described with reference to FIG. 2. As shown in FIG. 2(a), the image sensor 101 has a polarizing filter for each pixel. The polarizing filter of each pixel has a different polarization direction than the polarizing filter of an adjacent pixel, allowing adjacent pixels to photoelectrically convert polarized light of different polarization directions. Hereinafter, the pixel section 112 with polarizing filters arranged in this manner will be referred to as the "image sensor plane polarization sensor 112."

[0019] As shown in Fig. 2(b), four types of polarizing filters with polarization directions differing by 45° are arranged in the imaging plane polarization sensor 112, with each set consisting of four pixels. As an example, polarizing filter 113 with a polarization angle of 0°, polarizing filter 114 with a polarization angle of 45°, polarizing filter 115 with a polarization angle of 90°, and polarizing filter 116 with a polarization angle of 135° are arranged. Hereinafter, this set of four pixels will be referred to as a "local four pixel." A large number of local four pixels are arranged periodically in the imaging plane polarization sensor 112, as shown in Fig. 2(a).

[0020] With this configuration, polarized light with multiple different polarization directions can always be photoelectrically converted at the same time and captured as an image of the same frame. With this image sensor 101, it is possible to capture images using polarized light with four different polarization directions in a single capture. In this way, by using the image-capturing polarization sensor 112 with the polarization filter shown in Figure 2 arranged on the imaging surface, it is no longer necessary to arrange multiple PL filters with polarization properties for different polarization directions in front of the lens and move them manually or automatically in order to obtain images using polarized light with different polarization directions.

[0021] If the imaging plane polarization sensor 112 does not have a spectroscopic function such as a color filter, the image obtained will be monochrome, but it is possible to obtain a color image by arranging color filters of the same color for every four local pixels, as shown in Fig. 3. Here, as shown in Fig. 3(b), four local pixels covered with red (R) color filters 118, four local pixels covered with green (G) color filters 119, and four local pixels covered with blue (B) color filters 120 are arranged in a Bayer array as shown in Fig. 3(a).

[0022] 2 and 3 show an example of the imaging plane polarization sensor 112 that constitutes the image sensor 101, and the arrangement and polarization direction of the polarization filter can be set arbitrarily. It is also possible to combine the use of a polarization filter with or without a polarization filter.

[0023] Here, the relationship between the polarization direction of polarized light and the brightness of the output image signal will be explained using an example of photographing the subject shown in Figure 4. By extracting only the signals of pixels where a polarizing filter of a certain polarization direction is arranged from the image signals of a single image photographed by imaging plane polarization sensor 112 and generating a single image, it is possible to obtain an image using polarized light with the same polarization direction as that polarizing filter. By performing this for each of polarizing filters 113 to 116, images with four different polarization directions can be obtained.

[0024] 4 shows a subject in which a person 402 is inside a car 401, and the windshield of the car is taken as an area 400. Here, it is assumed that the amount of reflected light in the area 400 is large. The polarization characteristics of the reflected light will be described in detail later.

[0025] FIG. 5 shows four example polarized images obtained by separating the image signals obtained from the image sensor 101 according to the polarization direction. FIG. 5(a) shows polarized image 121 generated from image signals output only from pixels in which the polarizing filter 113 with a polarization angle of 0° shown in FIG. 2 is disposed. FIG. 5(b) shows polarized image 122 generated from image signals output only from pixels in which the polarizing filter 114 with a polarization angle of 45° is disposed. FIG. 5(c) shows polarized image 123 generated from image signals output only from pixels in which the polarizing filter 115 with a polarization angle of 90° is disposed. FIG. 5(d) shows polarized image 124 generated from image signals output only from pixels in which the polarizing filter 116 with a polarization angle of 135° is disposed. In FIG. 5, areas in which light reflected from the subject passes through the polarizing filter appear bright, and areas in which light is blocked by the polarizing filter appear dark.

[0026] Here, polarizing filter 114 with a polarization angle of 45° transmits the most reflected light from the windshield, so region 400 shown in Figure 5(b) is expressed as bright. Polarizing filter 116 with a polarization angle of 135° blocks the most reflected light from the windshield, so region 400 shown in Figure 5(d) is expressed as dark. Furthermore, polarizing filter 113 with a polarization angle of 0° and polarizing filter 115 with a polarization angle of 90° transmit the amount of reflected light intermediate between polarizing filter 114 and polarizing filter 116, so region 400 shown in Figures 5(a) and 5(c) is expressed with intermediate brightness.

[0027] Furthermore, areas in the four-directional polarized image where the brightness does not change, i.e., areas other than area 400, are non-polarized areas where the light is not polarized. In other words, since the polarization angle cannot be adjusted in the non-polarized areas, the only area in the subject example shown in Figure 4 where the magnitude of reflected light can be adjusted is area 400.

[0028] A digital signal (image signal) generated by the imaging plane polarization sensor 112 in the imaging element 101 is input to the image acquisition unit 102. The image acquisition unit 102 receives the image signal from the imaging element 101 and outputs it to the region determination unit 103 and the approximate function calculation unit 104.

[0029] The region determination unit 103 aggregates the polarization characteristics acquired by the four local pixels when performing a polarization removal process (described later) on the image signal input from the image acquisition unit 102, and determines a range (statistical region) for creating a histogram. For example, if the difference between the maximum and minimum values ​​of the four image signals (luminance values) output from each of the four local pixels is equal to or greater than a predetermined threshold, the region determination unit 103 determines that the four local pixels belong to a region where polarization removal is to be performed (hereinafter referred to as a "polarization removal target region"). The region determination unit 103 then determines the polarization removal target region thus obtained as the statistical region. Note that the statistical region may be smaller than the polarization removal target region. The approximate function calculation unit 104 calculates the polarization characteristics of polarized light for every four local pixels from the image signal acquired by the image acquisition unit 102. Here, a method for calculating the polarization characteristics will be described.

[0030] FIG. 6(a) is a graph of the approximation function F(θ) representing the polarization characteristics of any four local pixels within the region 400. In this embodiment, the component of the intensity I(θ) that changes for each polarization angle θ is called the "polarization component." When the intensity I(θ) at polarization angles of 0°, 45°, 90°, and 135° is acquired from the image obtained by the image acquisition unit 102 and the acquired intensity I(θ) is plotted for each polarization angle θ, the intensity I(θ) can be approximated by a sine function or cosine function with a period of 180°. In other words, the approximation function F(θ) can be obtained. Note that in this embodiment, the intensity at four polarization angles is used using the polarization filters 113-116 shown in FIG. 2 . However, if the intensity at three or more polarization angles is known, the approximation function F(θ) of the polarization component can be generated.

[0031] The approximate function F(θ) of this polarization component can be expressed by equation (1). F(θ)=Acos(2θ+2B)+C …(1) In equation (1), A is the polarization intensity of the polarized component, B is the polarization angle of the polarized component, and C is the offset component. In the approximation function F(θ), the point with minimum brightness Imin has the least polarized component, and therefore represents the brightness of only the unpolarized component. By calculating this approximation function F(θ), it is possible to adjust the polarization angle θ so that the magnitude of the reflected light is the desired magnitude.

[0032] In the example of this embodiment shown in FIG. 6(a), the polarized light component in the region 400 has a maximum luminance Imax when the polarization angle is 45°, and a minimum luminance Imin when the polarization angle is 135°, as described above.

[0033] 6(b) shows the polarization component (non-polarized component) of the non-polarization region excluding region 400 in FIG. 5, where the polarization component is 0 at all polarization angles θ. In reality, there are few scenes in which the polarization component is completely 0, but in this embodiment, for the sake of convenience, the polarization component is set to 0.

[0034] The polarization intensity extraction unit 105 extracts intensity information of the polarization component (polarization intensity) from the approximation function F(θ) of the polarization component calculated by the approximation function calculation unit 104, and outputs it to the polarization intensity histogram generation unit 106. The polarization intensity refers to the amplitude of the approximation function F(θ), and the polarization intensity changes depending on the brightness of the polarized light. The polarization intensity histogram generating unit 106 collects the polarization intensities for each of the local four pixels extracted by the polarization intensity extracting unit 105 in the region determined by the region determining unit 103, and creates a histogram of occurrence frequency. The polarization intensity mode calculation unit 107 calculates the polarization intensity (representative intensity) that is the most frequent value in the histogram generated by the polarization intensity histogram generation unit 106 and outputs it to the polarization removal processing unit 111 .

[0035] Meanwhile, the polarization angle extraction unit 108 extracts angle information (polarization angle) of the polarization component from the approximation function F(θ) of the polarization component calculated by the approximation function calculation unit 104, and outputs it to the polarization angle histogram generation unit 109. The polarization angle refers to the angle at which the approximation function F(θ) shows the maximum brightness Imax. The polarization angle histogram generating unit 109 tally up the polarization angles for every local four pixels extracted by the polarization angle extracting unit 108 within the statistical region determined by the region determining unit 103, and creates a histogram of occurrence frequency. The polarization angle mode calculation unit 110 calculates the polarization angle (representative angle) that is the most frequent value in the histogram generated by the polarization angle histogram generation unit 109 and outputs it to the polarization removal processing unit 111 .

[0036] The polarization removal processor 111 determines representative polarization characteristics within the statistical domain using the mode of polarization intensity obtained by the polarization intensity mode calculator 107 and the mode of polarization angle obtained by the polarization angle mode calculator 110. Then, the polarization components at each angle (here, 0°, 45°, 90°, and 135°) are calculated from the determined representative polarization characteristics, and the polarization components are subtracted from each brightness value. For example, by substituting the mode of polarization intensity Imode into A, the mode of polarization angle θmode into B, and each angle into θ in equation (1), an approximation function of the polarization components shown in equation (2) below can be obtained. Then, by substituting each polarization angle into equation (2) below, the polarization components at each polarization angle are calculated.

[0037] F(θ)=Imode·cos(2θ+2θmode)+C …(2)

[0038] Then, the polarized light component can be removed by subtracting the obtained polarized light component from the brightness I(θ) obtained from each pixel in the polarized light removal target area.

[0039] Next, we will explain the above-mentioned process for removing the polarized light component and why it is possible to properly remove the polarized light component and generate a polarized light-removed image that maintains resolution even when the edge of a person inside a vehicle overlaps with the polarized light component, with reference to the flowchart in Figure 7. Hereinafter, edges due to unpolarized light components, such as a person inside a vehicle, will be referred to as "unpolarized light component edges," and edges due to polarized light components, such as reflected light, will be referred to as "polarized light component edges."

[0040] First, in S101, an image signal is acquired from the imaging plane polarization sensor 112. In S102, the approximate function calculation unit 104 calculates an approximate function F(θ) of the polarization component for each local four pixels based on the image signal acquired by the image acquisition unit 102. Similar approximate functions F(θ) of the polarization component are calculated for regions with the same polarization characteristics. However, as shown in FIG. 4, if there is an edge of the non-polarized component, such as a person, behind the reflective windshield, the polarization characteristics represented by the approximate function F(θ) calculated for the local four pixels in region 400 that include the edge of the non-polarized component will differ from the original polarization characteristics. On the other hand, the approximate function F(θ) calculated for the local four pixels in region 400 that do not include the edge of the person will be an approximate function that shows the correct polarization characteristics.

[0041] In S103, the polarization intensity extraction unit 105 and the polarization angle extraction unit 108 calculate the polarization intensity and the polarization angle, respectively, from the polarization component approximation function F(θ) calculated for every four local pixels by the approximation function calculation unit 104.

[0042] In S104, a statistical region for acquiring a plurality of polarization characteristics is determined when the polarization elimination processor 111 removes the polarization component. As described above, the statistical region may be the polarization elimination target region (e.g., region 400) from which the polarization component is to be removed, or a smaller region within the polarization elimination target region.

[0043] In S105, the polarization intensities and polarization angles acquired in S103 are tallied for each statistical region determined in S104 by the region determination unit 103. Then, the polarization intensity histogram generation unit 106 and the polarization angle histogram generation unit 109 use the tallied polarization intensities and polarization angles to create histograms of occurrence frequency, respectively.

[0044] In S106, the polarization angle mode calculation unit 110 and the polarization intensity mode calculation unit 107 obtain the modes (representative intensity and representative angle) in the histograms of polarization intensity and polarization angle created in S105, respectively, and use these as the representative polarization characteristics of the statistical region.

[0045] When focusing on the region within region 400 that includes the edge of a person, the approximation function F(θ) obtained from the four local pixels without edges of the unpolarized component behind the windshield is a function that represents the correct polarization characteristics, and this polarization characteristic is the most abundant component within the statistical region. On the other hand, the approximation function F(θ) calculated from the four local pixels that include edges of the unpolarized component differs from the polarization characteristics of the four local pixels that do not include edges, and the approximation function F(θ) obtained varies depending on the angle and contrast of the edge. Therefore, the histogram generated within this region is a histogram that contains a mixture of correct polarization characteristics and randomly distributed polarization characteristics that include edges, as shown in Figures 8(a) and 8(b). Therefore, by calculating the most frequent values ​​of the polarization intensity and polarization angle in the statistical region from the generated histogram and using them as the representative polarization characteristics, the influence of the edge components can be removed.

[0046] In S107, the representative polarization characteristics acquired in S106 are used to calculate the polarization component for each polarization angle as explained in equation (2), and the polarization component is reduced by subtracting it from the brightness value of each pixel.

[0047] As described above, according to the first embodiment, the polarization component removal process (correction process) is performed on all pixels in the polarization removal target region using the representative polarization characteristics of the corresponding statistical region. This makes it possible to more accurately reduce polarization components by applying polarization characteristics that are not affected by edges to local four pixels whose calculated polarization characteristics are affected by edges when the local four pixels include edges. Furthermore, by subtracting an appropriate amount of polarization components for each polarization angle of the polarizing filter from the brightness value of each pixel, it becomes possible to reduce polarization components without reducing resolution.

[0048] <Second embodiment> Next, a second embodiment of the present invention will be described. In the first embodiment described above, the representative polarization characteristics within the statistical region are calculated, and an appropriate amount of polarization component for each polarization angle is subtracted from the brightness values ​​of the corresponding pixels in the polarization removal target region, thereby reducing the polarization component while maintaining resolution. However, as with the subject shown in Figure 9, the light reflected from the polarization removal target region is not necessarily limited to only one type of polarization component. Depending on the object reflecting the light, the shape of the histogram may vary significantly; for example, multiple peaks may appear in the polarization intensity histogram.

[0049] Therefore, in the second embodiment, a method for removing polarized light components from an object that reflects light as shown in Fig. 9 will be described using the flowchart in Fig. 10. Area 901 in Fig. 9 is an area with reflected light having low brightness, and area 902 is an area with reflected light having high brightness. Note that in Fig. 10, the same steps as those shown in Fig. 7 are assigned the same step numbers, and descriptions thereof will be omitted where appropriate.

[0050] In a subject containing multiple reflected light beams with different polarization intensities, as shown in Figure 9, the polarization removal target area may contain a boundary between polarization beams. In this case, if a histogram of polarization intensity is created in S105, the histogram will contain multiple peaks. In this case, in the method of the first embodiment, in S106, the most frequently occurring characteristic is selected as the mode within the statistical area as the representative polarization characteristic, and in S107, polarization removal is performed for each polarization removal target area using the representative polarization characteristic. However, if the polarization removal target area contains multiple polarization components with different polarization characteristics, the polarization components cannot be removed correctly, which can cause noise.

[0051] Because the polarization intensities are different between regions 901 and 902, the amplitudes of the approximation function F(θ) are different. Therefore, if a statistical region straddles regions 901 and 902, two types of polarization intensities will be included within the statistical region. If a histogram is created using such a statistical region, two peaks corresponding to regions 901 and 902 will appear in the polarization intensity histogram, as shown in Figure 8(c). In this case, the peak with the larger value is selected as the mode and becomes the representative polarization characteristic of the statistical region. However, if the polarization characteristic of region 901 is selected as the representative polarization characteristic and polarization components are removed, the incorrect polarization component will be removed from region 902.

[0052] Therefore, in the second embodiment, a plurality of polarization-removed images in which the polarization component has been removed are generated while changing the statistical region of the polarization component. In S208, it is determined whether a predetermined number of polarization-removed images corrected by changing the statistical region have been created through the processes of S101 to S107. If the predetermined number of times has been reached, the process proceeds directly to S210. On the other hand, if the predetermined number of times has not been reached, the process proceeds to S209, where the statistical region is changed.

[0053] For a subject like the one shown in Figure 9, depending on the statistical region pattern, it may contain multiple polarized light components with different polarization intensities, such as regions 901 and 902, resulting in regions where the polarization component cannot be successfully removed. In such cases, by creating multiple polarization-removed images with different statistical region positions and sizes, a statistical region is selected based on the statistical region pattern so that it does not include the boundaries of the polarized light components, and a corrected image is created.

[0054] In S210, multiple depolarized images created by changing the statistical area are combined, such as by averaging, to create a single depolarized image. In this way, noise may occur in each depolarized image due to depolarization of an incorrect polarization component, but by combining multiple images with different depolarization target areas, the effect of noise can be reduced and the resulting image can approach the correct depolarized image.

[0055] In the second embodiment, multiple images are created by changing the size and position of the statistical region and removing the polarization component, and then a synthesis process such as averaging is performed. In other words, even if two types of polarized light are included in the region near the boundary between two types of reflected light when the statistical region is taken the first time, only one type of polarized light may be included in the statistical region when the statistical region is taken the Nth time. By synthesizing multiple polarization-removed images corrected based on the representative polarization information of different statistical regions obtained in this way, noise that occurs when polarization is removed using an incorrect value is reduced compared to images where polarization is removed using a single polarization removal target region.

[0056] As described above, according to the second embodiment, by combining multiple images that have been depolarized using statistical regions of different sizes and positions, it is possible to perform polarization removal (correction processing) suited to the subject or scene.

[0057] <Third embodiment> Next, a third embodiment of the present invention will be described. In the second embodiment, a method for removing polarized light components by changing the size and position of the statistical region when the region to be removed contains multiple types of reflected light with different polarization intensities was described. In the third embodiment, a method for removing another polarized light component when the region to be removed contains multiple types of reflected light with different polarization intensities, similar to the second embodiment, will be described.

[0058] Fig. 11 is a flowchart showing a method for removing a polarized light component in the third embodiment. In Fig. 11, the same steps as those shown in Fig. 7 are given the same step numbers, and descriptions thereof will be omitted where appropriate.

[0059] In S306, the number of occurrence frequency peaks is obtained in the histogram of polarization intensity obtained in S105. For example, if the statistical region of the object shown in Figure 9 includes regions 901 and 902, two types of polarization intensity are included, and therefore two peaks corresponding to regions 901 and 902 occur in the histogram of polarization intensity, as shown in Figure 8(c). On the other hand, if the statistical region is region 400 as shown in Figure 4, the histogram of polarization intensity will have one peak.

[0060] In S307, it is determined whether the histogram of polarization intensity contains multiple peaks. If there is one peak, in S106 and S107, the same processing as that described in the first embodiment with reference to FIG. 7 is performed, and polarization removal is performed using the representative polarization characteristic of the statistical region.

[0061] On the other hand, if multiple peaks are included, the process proceeds to S308, where the polarization components are calculated based on the polarization intensity and polarization angle calculated for each of the four local pixels, and the polarization components are removed.

[0062] As described above, according to the third embodiment, when there is one peak in the histogram of polarization intensity within the statistical region, polarization removal is performed using the representative polarization characteristics for the image signal within the polarization removal target region. On the other hand, when there are multiple peaks, polarization components are removed (corrected) using the polarization intensity and polarization angle calculated for every four local pixels.

[0063] Thus, according to the third embodiment, even when multiple polarization components are included within the polarization removal target area and it is not possible to identify a single representative polarization characteristic, it is possible to suppress the generation of noise caused by the removal of an incorrect polarization component.

[0064] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. In the third embodiment, when it is determined that the histogram of polarization intensity in the region to be polarized light removed contains multiple peaks, the polarization characteristics calculated for each local four-pixel region are used to remove the polarization component for each local four-pixel region, thereby avoiding the removal of polarization due to incorrect polarization components. However, the histogram of polarization intensity does not necessarily have a shape that allows multiple peaks to be identified, as in Figure 8(c). Depending on the subject, the peaks may not be clear, as in Figure 8(d), making it difficult to accurately determine the number of peaks.

[0065] Therefore, in the fourth embodiment, a method for improving the accuracy of determining the number of peaks by excluding unnecessary polarization characteristics when the histogram of polarization intensity has a shape as shown in FIG. 8(d) will be described.

[0066] If the local four pixels for which the approximation function is calculated contain edges of polarized or unpolarized components, it is generally impossible to calculate the correct polarization characteristics. Furthermore, the calculated approximation function varies depending on the edge orientation and the polarization characteristics included. For example, if the area to be depolarized contains many edges of unpolarized components, creating a histogram of polarization intensity will result in a histogram with one peak and a shape in which noise components containing edges within the local four pixels are dispersed, as shown in Figure 8(d). Furthermore, if the area to be depolarized contains multiple different types of polarization intensity, creating a histogram will result in the number of peaks varying depending on the type of polarization intensity. However, depending on the shape of the peak, it may be impossible to determine the number of peaks because they are buried in noise components.

[0067] Therefore, in the fourth embodiment, the mode is obtained from the polarization angle histogram, and the polarization intensities of four local pixels with polarization angles far from the mode are excluded from the polarization intensity histogram. That is, the polarization intensity histogram is regenerated using the polarization intensities of four local pixels with polarization angles within a predetermined range from the mode. This makes it possible to create a polarization intensity histogram that excludes pixels that include edges within the four pixels and for which an approximation function of an incorrect polarization characteristic has been calculated. This makes it possible to create a polarization intensity histogram that includes only correct polarization characteristics, thereby improving the determination accuracy in the third embodiment.

[0068] <Fifth embodiment> Next, a fifth embodiment of the present invention will be described. By acquiring image signals in which exposure conditions are changed for four local pixels with different polarization directions, it is possible to perform HDR (High Dynamic Range Rendering), a method for expanding the dynamic range with a single exposure. FIG. 12 is a diagram showing the relationship between the arrangement of polarizing filters and exposure time. In this embodiment, the polarizing filters 113 to 116 are arranged in the same manner as those shown in FIG. 2, and the exposure time of the pixels corresponding to the polarizing filters 113 and 115 is set to long seconds (L), and the exposure time of the pixels corresponding to the polarizing filters 114 and 116 is set to short seconds (S). Then, by performing image composition taking into account the ratio of each exposure, it is possible to compose an image with a wide dynamic range.

[0069] Here, a simple HDR compositing method without using a polarizing filter will be described with reference to FIG. 13. Here, the long-time (L) exposure condition is referred to as the high-gain signal, and the short-time (S) exposure condition is referred to as the low-gain signal. The solid line in FIG. 13(a) shows the relationship between the input and output luminance values ​​of the low-gain signal. The dashed line in FIG. 13(b) shows the relationship between the input and output luminance values ​​of the high-gain signal. The solid line in FIG. 13(b) shows the relationship between the input and output luminance values ​​of the converted high-gain signal, which is obtained by compressing the relationship between the input and output luminance values ​​of the high-gain signal based on the ratio of the exposure conditions of the high-gain signal and the low-gain signal. Finally, FIG. 13(c) shows the low-gain signal shown in FIG. 13(a) and the converted high-gain signal shown in FIG. 13(b) on the same graph. For example, HDR compositing is performed by generating output luminance values ​​using the converted high-gain signal in input luminance value range (i) and generating output luminance values ​​using the low-gain signal in input luminance value range (ii).

[0070] However, when polarizing filters 113 to 116 with different polarization angles are placed in front of each of the four local pixels, the ratio of pixel values ​​and the ratio of exposure conditions do not match between low-gain signals and high-gain signals.

[0071] Figure 14 shows an example of the relationship between input and output luminance values ​​when polarizing filters with different polarization angles are used and the HDR compositing method shown in Figure 13 is applied. The dotted line in Figure 14(a) shows the relationship between the input and output luminance values ​​of the low-gain signal shown by the solid line in Figure 13(a), and the solid line in Figure 14(a) shows the relationship between the input and output luminance values ​​of the low-gain signal that has passed through a polarizing filter with a predetermined polarization angle. The dashed line in Figure 14(b) shows the relationship between the input and output luminance values ​​of the high-gain signal shown by the dashed line in Figure 13(b), and the dotted line in Figure 14(b) shows the relationship between the input and output luminance values ​​of the high-gain signal that has passed through a polarizing filter with a predetermined polarization angle. Furthermore, the solid line in Figure 14(b) shows the relationship between the input and output luminance values ​​of the converted high-gain signal after passing through a polarizing filter that has been converted based on the ratio of the exposure conditions for the high-gain signal and the low-gain signal.

[0072] FIG. 14(c) shows on the same graph the low-gain signal after passing through the polarizing filter, indicated by the solid line in FIG. 14(a), and the converted high-gain signal after passing through the polarizing filter, indicated by the solid line in FIG. 14(b). For example, in input luminance value region (i), the output luminance value is generated using the converted high-gain signal after passing through the polarizing filter, and in input luminance value region (ii), the output luminance value is generated using the low-gain signal after passing through the polarizing filter. However, in this case, a step occurs at the boundary between input luminance value region (i) and input luminance value region (ii). Furthermore, because the slope of the low-gain signal after passing through the polarizing filter differs from the slope of the converted high-gain signal after passing through the polarizing filter, linearity is not maintained. Therefore, there is a problem that the desired HDR synthesis cannot be performed due to the step and nonlinearity caused by errors in the polarizing filter.

[0073] In view of the above-mentioned problems, in the fifth embodiment, HDR synthesis will be described in the case where the polarization removal method implemented in the first embodiment is performed.

[0074] 15 is a block diagram showing the functional configuration of an image capture device 500 according to the fifth embodiment, which performs HDR synthesis after polarization removal. The image capture device 500 is configured by adding a gain conversion unit 501 and an HDR synthesis unit 502 to the image capture device 100 shown in FIG. 1, and other components are the same as those of the image capture device 100. Therefore, the same reference numerals are used and descriptions thereof are omitted.

[0075] In the fifth embodiment, as shown in Fig. 12, every four local pixels are covered with polarizing filters 113 to 116, and the pixels covered with polarizing filters 113 and 115 are exposed for a long time (L), while the pixels covered with polarizing filters 114 and 116 are exposed for a short time (S). Note that in the fifth embodiment, it is assumed that high-gain signals and low-gain signals are obtained depending on the exposure time, but the present invention is not limited to this. For example, the sensitivity may be changed using the gain of an amplifier included in the image sensor 101, or both the exposure time and the sensitivity may be changed.

[0076] Image signals acquired by the image sensor 101 and having different exposure conditions within four local pixels are sent to the image acquisition unit 102 and then input to the gain conversion unit 501. An example of pixel values ​​(luminance values) obtained from the four local pixels input to the gain conversion unit 501 is shown in FIG. 16. A high-gain signal, luminance I H (0°). From the pixel covered by the polarizing filter 114 at a polarization angle of 45°, a low gain signal, luminance I L (45°). A high-gain signal, intensity I H (90°). A low-gain signal, luminance I L (135°) is obtained.

[0077] Since the exposure conditions are different for the four local pixels, if the output of the image acquisition unit 102 is used as is to remove the polarization, it is not possible to acquire appropriate polarization characteristics. Therefore, the gain conversion unit 501 converts the values ​​acquired at the same gain for each pixel based on the exposure conditions for the four local pixels. Here, the luminance I acquired with a high gain signal is converted to a value obtained at the same gain for each pixel. H (0°) and luminance I H (90°) is converted to a low-gain signal, and the pixel value of the converted high-gain signal is the luminance I L (0°) and luminance I L However, the gain conversion method is not limited to this, as the gain conversion may be adjusted to a high gain signal or to another gain.

[0078] The image signal converted by the gain conversion unit 501 is depolarized by processing in the polarization removal processing unit 111 from the region determination unit 103 described in the first embodiment, and the depolarized image is input to the HDR synthesis unit 502. If appropriate polarization removal is performed on the polarization-removed image input to the HDR synthesis unit 502, the relationship shown in FIG. 13(c) can be obtained. Therefore, as described in FIG. 13, HDR synthesis is performed by using a converted high-gain signal in region (i) and a low-gain signal in region (ii). However, the HDR synthesis method described in this embodiment is merely an example and is not limited to this. For example, a low-gain signal may be converted and used for HDR synthesis, or pixel values ​​for each exposure condition may be appropriately selected or added.

[0079] Next, a process for performing HDR combining after removing the polarization component in the fifth embodiment will be described with reference to the flowchart shown in Fig. 17. Note that in the process in Fig. 17, the same processes as those described in the first embodiment with reference to Fig. 7 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0080] In S501, the image acquisition unit 102 acquires image signals obtained under different exposure conditions from each of the four local pixels output from the image sensor 101. In S502, the acquired image signals are subjected to gain conversion in the gain conversion unit 501 as described above, and are adjusted so that differences due to exposure conditions in the acquired image signals are uniform, thereby converting them into image signals that can remove polarization.

[0081] Then, in S102 to S106, polarization removal processing (correction processing) is performed using the methods described in the first to fourth embodiments based on the polarized image that has been gain-converted by the gain conversion unit 501. In S503, HDR synthesis is performed in the HDR synthesis unit 502 using the image signal from which the polarization component has been removed by the polarization removal processing unit 111.

[0082] As described above, according to the fifth embodiment, image signals with different exposure conditions are acquired for four local pixels, and HDR compositing is performed after aligning the exposure conditions and removing polarization. As a result, even if the four local pixels are covered with polarizing filters with different polarization angles and different exposure conditions are used for the four local pixels to perform HDR compositing, it is possible to generate an HDR image with an expanded dynamic range while reducing the influence of pixel value errors due to the polarizing filters.

[0083] Sixth Embodiment Next, a sixth embodiment of the present invention will be described. Note that the configuration of the image capture device 100 in the sixth embodiment is the same as that shown in Fig. 1, and therefore a description thereof will be omitted. As explained in the embodiments so far, the image capture plane polarization sensor 112 is provided with polarization filters with multiple different polarization angles, with each set consisting of four local pixels, and the incident light is photoelectrically converted and output as an image signal. The brightness value of the output image signal is then used to find an approximation function of the brightness component (polarization component) that changes depending on the polarization angle, and the polarization component is extracted.

[0084] However, if the edge of the subject falls within a local 4-pixel, it may not be possible to obtain a correct approximation function for that local 4-pixel, or even if an approximation function is obtained, the polarization component for that local 4-pixel will be completely different from that of the surrounding area, resulting in false information.For this reason, the polarization component for each polarization angle in that statistical area was obtained from the most frequent value of the polarization information within the polarization removal target area, and this was applied to all local 4-pixels within the polarization removal target area.

[0085] In contrast, in the sixth embodiment, the polarization components of the four local pixels are first compared with the representative polarization component in the statistical domain to determine whether the polarization intensities or polarization angles are similar. The processing in this embodiment will be described below with reference to the flowchart in FIG.

[0086] First, in S601, the target image signal is input from the imaging plane polarization sensor 112, and in S602, the most frequent values ​​of the polarization intensity and polarization angle in the statistical region are obtained as described in any of the first to fourth embodiments.

[0087] Then, in S603, the intensity difference and angle difference between the polarization intensity and polarization angle of each of the four local pixels and the representative intensity and representative angle of the statistical region are calculated.

[0088] In S604, it is determined whether the intensity difference and angle difference calculated in S603 are equal to or greater than their respective preset thresholds. If they are less than the thresholds, it is determined that no edge portion is included, and the process proceeds to S605, where polarization removal is performed on the image signals output from each of the four local pixels using the polarization components calculated in S602.

[0089] On the other hand, if at least one of the intensity difference and angle difference is equal to or greater than the threshold, it is determined that the local four pixels contain an edge portion, and the process proceeds to S606, where the grouping of the local four pixels is changed. Here, as shown in FIG. 19, new local four pixels are formed by shifting one pixel in either the up, down, left, or right direction. Then, in S607, an approximation function F(θ) of the new local four pixels is calculated, and the polarization intensity and polarization angle are obtained. Then, in S608, the intensity difference and angle difference between the polarization intensity and polarization angle of the new local four pixels and the representative intensity and representative angle are calculated.

[0090] In S609, it is determined whether the intensity difference and angle difference calculated in S608 are equal to or greater than their respective thresholds. If both are less than the thresholds, it is determined that the new local four pixels do not contain an edge portion, and the process proceeds to S610, where polarization removal is performed on the image signal output from the new local four pixels using the polarization component calculated in S602.

[0091] On the other hand, if at least one of the intensity difference and angle difference is greater than or equal to the threshold, it is determined that the new local four pixels also contain an edge portion, and the process returns to S606, where the grouping of the local four pixels is changed to another grouping, and the above-mentioned process is repeated.

[0092] If the difference is equal to or exceeds the threshold value regardless of whether the grouping of the local four pixels is changed in the up, down, left, or right direction, polarization removal is performed on the grouping of the local four pixels with the smallest difference, for example.

[0093] As described above, according to the sixth embodiment, by changing the combination of the local four pixels to create local four pixels that do not include edges, it is possible to perform polarization removal suitable for various subjects. <Modification of the Sixth Embodiment> In the sixth embodiment, a case where a combination of four local pixels is shifted and reformed has been described, but in this modified example, processing when color filters of multiple colors are arranged as shown in Fig. 3 will be described using the flowchart in Fig. 20. Note that processing similar to the processing shown in Fig. 18 described in the sixth embodiment is given the same reference numerals, and description thereof will be omitted as appropriate.

[0094] The image signal input by S601 is sent to a polarization sensor with a color filter. 112 3A and 3B, each of the four local pixels is grouped by color (R, G, B), and the polarization intensity and polarization angle are calculated for each.

[0095] In S604, if the intensity difference and angle difference calculated in S603 are equal to or greater than predetermined thresholds, it is determined that the object of the local 4-pixel contains an edge portion, and the process proceeds to S606, where a new local 4-pixel is formed using adjacent pixels.

[0096] In this modification, color filters of the same color are arranged for each of the four local pixels, so that in an image that has not yet been developed, the output for each color is different. For example, if the grouping of the four local pixels is shifted downward by one pixel as shown in Figure 21, the G filter and the B filter are mixed, and the approximation function F(θ) cannot be found from the output of the imaging plane polarization sensor 112 as is.

[0097] Therefore, in this modification, in S611, a white balance gain for polarization calculation is calculated from the representative intensity calculated within the statistical region to match the polarization intensity of each of R, G, and B to G. Then, the white balance gain for polarization calculation is multiplied by the polarization intensity of R and the polarization intensity of B, respectively, to make them approximately the same as G, and then an approximation function F(θ) is calculated using new local four pixels.

[0098] Conventionally, white balance gain is a gain value that adjusts the R and B levels to the standard white level, taking the G output of the standard white area as the reference. However, since the objective here is not to accurately represent the color of the subject, a white balance gain for polarization calculation is used to find an approximation function.

[0099] By performing the above-described processing, polarization removal (correction processing) suitable for various subjects can be performed even with an imaging plane polarization sensor provided with a color filter.

[0100] Seventh Embodiment Next, a seventh embodiment of the present invention will be described. In the seventh embodiment, a method will be described in which the type of edge in an image acquired by the imaging plane polarization sensor 112 is determined, and polarization removal (correction processing) is performed according to the determined type of edge.

[0101] Fig. 22 is a block diagram showing the functional configuration of an image capturing device 700 according to the seventh embodiment. The same components as those in Fig. 1 are given the same reference numerals, and descriptions thereof will be omitted where appropriate. The image capturing device 700 is a device capable of performing operations from image input to image output.

[0102] The image generation unit 703 separates the image signal acquired from the image sensor 101 by the image acquisition unit 102 for each polarization angle to generate the respective images. For example, when the subject shown in FIG. 4 is photographed, four polarization images 121 to 124 shown in FIG. 5 are obtained. In this case, the resolution of the polarization images generated by the image generation unit 703 is ¼ of the resolution of the image signal acquired from the image sensor 101 by the image acquisition unit 102.

[0103] In this embodiment, an example has been shown in which the image generation unit 703 generates an image by separating the image signal for each polarization angle, but an image with the same resolution as the image signal acquired from the image sensor 101 may be generated by interpolating using image signals output from surrounding pixels with the same polarization angle. Fig. 23 is a diagram for explaining the interpolation method, and shows polarizing filters arranged in the same order as the polarizing filters shown in Fig. 2(a).

[0104] Specifically, an image is generated by interpolating the pixel value of a polarization angle of 0° at a pixel of interest 711, which is provided with a polarizing filter 114 having a polarization angle of 45°, by averaging the pixel values ​​of surrounding pixels 712 and 713, which are provided with polarizing filters 113 having a polarization angle of 0°. However, the above interpolation method is just one example, and the interpolation method is not limited to this; it is also possible to refer to four surrounding pixels or pixels in diagonal directions.

[0105] Alternatively, the image sensor 101 may use the color imaging plane polarization sensor shown in Fig. 3. Fig. 24 is a diagram for explaining the interpolation method in this case, showing polarization filters arranged in the same order as the polarization filters shown in Fig. 3(a). In this case, first, interpolation is performed using the pixel value to be interpolated for the pixel of interest and surrounding pixels of the same color and polarization angle. Next, separation is performed for each polarization angle to generate plain images for each color and polarization angle.

[0106] Specifically, for a pixel of interest 721, which is provided with a B color filter and a polarizing filter 114 with a polarization angle of 45°, the G pixel value is interpolated by averaging the pixel values ​​of four surrounding pixels 722 to 725, which are provided with a G color filter and a polarizing filter 114. After generating a plain image for each color using this method, the image is separated by polarization angle to generate a polarized image for each color and polarization angle. However, the above interpolation method is only an example, and the interpolation may refer to eight surrounding pixels or pixels in a diagonal direction, and is not limited to this. Furthermore, the image generation method described above is only an example, and the generated image may be a G image only, or a YUV image obtained by YUV conversion of an RGB image, and the image generation method is not limited to this.

[0107] The edge detection unit 704 detects the presence or absence of an edge at a pixel of interest from the difference in brightness between the pixel and surrounding pixels in the polarization image generated by the image generation unit 703. Here, the edge detection method will be specifically described with reference to FIG.

[0108] FIG. 25 shows a polarized image generated by the image generating unit 703 at a polarization angle of 0°. StatueOf these, a total of 9 pixels are shown, 3 vertically and 3 horizontally. 1 A method for determining whether an edge exists will be described.

[0109] First, the average pixel value of the surrounding pixels 732, 735, and 737 is calculated and set as the first vertical average value. 、7 38 and pixel of interest 731 The average pixel value of the surrounding pixels 734, 736, and 739 is calculated and set as the second vertical average value. The average pixel value of the surrounding pixels 732, 733, and 734 is calculated in the same way in the horizontal direction and set as the first horizontal average value. Next, the average pixel value of the surrounding pixels 735, 736, and 739 is calculated and set as the third vertical average value. 、7 36 and pixel of interest 731 The average pixel value of the surrounding pixels 737, 738, and 739 is calculated and used as the third horizontal average value. Value and do.

[0110] Next, the difference between the first vertical average value and the second vertical average value is defined as the first difference. The difference between the second vertical average value and the third vertical average value is defined as the second difference. The difference between the first horizontal average value and the second horizontal average value is defined as the third difference. The difference between the second horizontal average value and the third horizontal average value is defined as the fourth difference. If any of the first to fourth differences is less than a predetermined threshold, it is determined that there is no edge, and if any of the first to fourth differences is equal to or greater than the predetermined threshold, it is determined that there is an edge.

[0111] However, the above-mentioned edge detection method is only an example, and the edge detection method itself is not limited to this, as it may be possible to perform edge detection in a diagonal direction or using a 3×3 edge detection filter.

[0112] The edge discrimination unit 705 determines the type of edge present in the local four pixels by comparing the detection results of the edge detection unit 704 with pixels at the same position in the polarization image generated by the image generation unit 703. The discrimination result of the edge discrimination unit 705 is either no edge, an edge in a non-polarized region where light is not polarized (a region other than region 400 in FIG. 4), an edge of a non-polarized component in a polarized region (region 400), or an edge of a polarized component in a polarized region. The edge discrimination method will be described in detail later.

[0113] The polarization characteristics processing unit 706 processes the polarization characteristics of the polarized image according to the determination result of the edge determination unit 705. Specifically, if the edge determination unit 705 determines that there is no edge, or that there is an edge in a non-polarized region, the polarization characteristics processing unit 706 calculates an approximation function F(θ) of the polarization characteristics from the local four pixels, and performs polarization removal for each local four pixel. Alternatively, polarization removal may be performed using the same method as in the first embodiment, or a conventional method.

[0114] Furthermore, if the edge determination unit 705 determines that the image is an edge of a non-polarized component in the polarization region, the polarization characteristics processing unit 706 performs the process described in the first embodiment, i.e., performs polarization removal using the representative polarization characteristics within the divided region. If the edge determination unit 705 determines that the image is an edge of a polarized component in the polarization region, the polarization characteristics processing unit 706 performs the process described in the second embodiment, i.e., performs polarization removal by combining multiple images with polarization removed in different region sizes. However, the process performed by the polarization characteristics processing unit 706 in this embodiment is just an example, and different processes may be performed depending on the result of the edge determination unit 705.

[0115] A method for determining the type of edge in a polarization image in the above-described system will be described using the flowchart in Fig. 26. The processing shown in Fig. 26 is performed by the image sensor 101 and edge determination unit 705 shown in Fig. 22.

[0116] First, in S701, an image signal is acquired from the image sensor 101. Then, in S702, an image generating unit 703 generates a plurality of polarization images for each polarization angle from the image signal acquired in S701. Next, in S703, the edge detection unit 704 performs edge detection on each of the polarization images generated in S702.

[0117] Then, in S704, the edge determination unit 705 determines whether or not there are any pixels determined to have an edge in the process of S703 in the edge detection process for the pixel signals output from the local four pixels 710 shown in Fig. 23. If there are no pixels determined to have an edge, the local four pixels 710 are determined to have no edge (S705), and the process proceeds to S711. If there are any pixels determined to have an edge, it is determined that an edge exists in the local four pixels 710, and the process proceeds to S706.

[0118] In S706, it is determined whether or not there are any pixels in which no edge has been detected among pixels 714-717 included in the local four pixel set 710. If it is determined that there is an edge in pixels of all polarization angles (NO in S706), the edge detected in the local four pixel set 710 is determined to be the edge of a non-polarized region (S707), and the process proceeds to S711. If it is determined that there is no edge in pixels of any polarization angle, the edge present in the local four pixel set 710 is determined to be the edge of a polarized region, and the process proceeds to S708.

[0119] In S708, the pixel values ​​of the pixels 714-717 included in the local four pixels 710, whose polarization angles are determined to have an edge, are compared with the pixel values ​​of the pixels whose polarization angles are determined to have no edge. If the pixel values ​​of the pixels whose polarization angles are determined to have an edge are equal to or less than the pixel values ​​of the pixels whose polarization angles are determined to have no edge, the edge present in the local four pixels 710 is determined to be an edge of the unpolarized component (S709), and the process proceeds to S711. On the other hand, if the pixel values ​​of the pixels whose polarization angles are determined to have an edge are greater than the pixel values ​​of the pixels whose polarization angles are determined to have no edge, the edge present in the local four pixels 710 is determined to be an edge of the polarized component (S710), and the process proceeds to S711.

[0120] In S711, it is determined whether the edge type determination process has been performed for all local four pixels, and if there are unprocessed local four pixels, the process returns to S704 and the above process is performed for the next local four pixels.

[0121] When the discrimination process is completed for all the local four pixels, the process of the edge discriminator 705 is completed.

[0122] As described above, according to the seventh embodiment, by comparing the edge detection results for each polarization angle, it is possible to determine the presence or absence of edges in a polarized image and the type of edge, and to perform polarization removal (correction processing) according to the subject and scene.

[0123] Eighth Embodiment Next, an eighth embodiment of the present invention will be described. Note that the configuration of the imaging device 100 in the eighth embodiment is the same as that shown in Fig. 1, and therefore a description thereof will be omitted.

[0124] In the first embodiment, the representative polarization characteristic of the polarization removal target area was calculated, and the correct amount of polarization components for each angle was subtracted from the polarization removal target area to remove the polarization components while maintaining resolution. However, when the polarization removal target area includes different reflective surfaces, uniformly removing the polarization components using the representative polarization characteristic within the statistical area results in good polarization removal for reflective surfaces with the same characteristics as the representative polarization characteristic, but insufficient polarization removal for reflective surfaces with characteristics different from the representative polarization characteristic. In other words, in order to achieve good polarization removal in a polarization removal method based on the representative polarization characteristic within the statistical area, it is desirable that most of the polarization characteristics in the polarization removal target area are the same as the representative polarization characteristic.

[0125] As an example, a method for removing polarization from an area that combines area 2701 of the windshield and area 2702 of the side window, such as the object shown in FIG. 27, will be described below. Area 2701 of the windshield and area 2702 of the side window have different reflective surfaces. FIG. 28 shows a histogram of polarization angles when the combined area of ​​areas 2701 and 2702 is used as the statistical area. In the polarization angle histogram, area 2701 has one peak 2801, while area 2702 has peak 2802 that is different from peak 2801 because the polarization angles of areas 2701 and 2702 are different. In this way, when a polarization angle histogram is created for a statistical area that includes different polarization characteristics, multiple peaks appear.

[0126] 27, the area of ​​region 2701 is larger than that of region 2702, and therefore the number of pixels included in region 2701 is also larger, resulting in a higher frequency of occurrence of polarization angles caused by region 2701. When the mode is calculated for this histogram of polarization angles, the peak corresponding to region 2701 is the mode, and polarization can be satisfactorily removed in region 2701. On the other hand, polarization components are removed at a different polarization angle for region 2702, and therefore polarization components cannot be sufficiently removed in region 2702.

[0127] Fig. 29 is a flowchart showing a process for solving the above problem. In Fig. 29, the same processes as those in Fig. 7 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0128] In S801, the polarization angle mode calculation unit 110 counts the number of peaks in the polarization angle histogram generated by the polarization angle histogram generation unit 109 and determines the number of reflecting surfaces included in the statistical region from the counted number of peaks. For example, if the number of peaks is two, it is determined that two reflecting surfaces with different polarization characteristics are included. If it is determined that there are multiple reflecting surfaces, the process returns to S104 and the size of the statistical region is re-determined. At this time, the size of the statistical region is set smaller than the size previously determined. By repeating this process, the statistical region will be composed of one reflecting surface. In S106, the mode is obtained from the histogram of polarization intensity and polarization angle generated in the statistical region, and in S107, polarization removal is performed within the polarization removal target region based on the obtained mode. In this case, it is preferable that the statistical region and the polarization removal target region are the same.

[0129] In S802, it is determined whether or not the polarization removal process has been performed for all polarization regions, and if there is a region where polarization removal has not been performed, the process returns to S104 and the above process is repeated for that region.

[0130] In a real image, even if the reflecting surface is uniform, multiple peaks may be formed in the histogram due to the edges of the subject or the edges of the reflected light, noise, etc. In such cases, the number of peaks may be determined by smoothing the histogram or by performing maximum likelihood estimation.

[0131] As described above, according to the eighth embodiment, even when the subject includes multiple reflecting surfaces, polarization removal (correction processing) can be performed satisfactorily.

[0132] <Ninth embodiment> Next, a ninth embodiment of the present invention will be described. Note that the configuration of the image pickup device 100 in the ninth embodiment is the same as that shown in Fig. 1, and therefore description thereof will be omitted. In the first embodiment, the representative polarization characteristics within the statistical region were calculated, and the correct amount of polarization component for each polarization angle was subtracted from the brightness value to remove the polarization component while maintaining resolution. However, when a curved surface is included in the statistical region, the histogram becomes smooth, and polarization removal is insufficient at the edge of the curved surface. For example, assume that the polarization angle histogram created for region 3001 of the windshield in Figure 30 is shown in Figure 31(a). Because the windshield is curved, the polarization angles calculated for every four local pixels within region 3001 are not uniform, and the histogram becomes smooth. In particular, the polarization characteristics differ at the edge of the windshield, where the curvature varies greatly. Even if polarization removal is performed based on the mode calculated from this histogram, satisfactory polarization removal is not achieved in areas outside the histogram peak, especially at the edge of the windshield.

[0133] Fig. 32 is a flowchart showing the processing in the ninth embodiment. In Fig. 32, the same processes as those in Fig. 7 are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0134] In S901, the polarization angle mode calculation unit 110 compares the variance of the polarization angle histogram generated by the polarization angle histogram generation unit 109 with a threshold. If the variance of the histogram is equal to or greater than the threshold, it determines that the statistical region includes a curved surface, and returns to S104 to re-determine the size of the statistical region. This time, the size is made smaller than the previously determined statistical region. By repeating this process, the statistical region becomes composed of reflective surfaces with similar curvatures. In S106, the mode is obtained from the polarization intensity histogram and polarization angle histogram generated for the statistical region, and in S107, polarization removal is performed within the polarization correction target region based on the obtained mode. In this case, it is preferable that the statistical region and the polarization removal target region are the same.

[0135] Region 3002 shown in Figure 30 is within region 3001, but is a smaller region. Figure 31(b) shows a histogram of polarization angles created for this region 3002. Although region 3002 is also within a curved surface, there is not much difference in the curvature of the reflecting surface within region 3002, so the kurtosis of the histogram increases. In this way, when the reflecting surface is curved, narrowing the statistical region can increase the kurtosis of the histogram, resulting in good polarization removal (correction processing).

[0136] In S902, it is determined whether or not the polarization removal process has been performed for all polarization regions, and if there is a region where polarization removal has not been performed, the process returns to S104 and the above process is repeated for that region.

[0137] In this embodiment, whether a statistical region contains a curved surface is determined by comparing the variance of the polarization angle histogram with a threshold value. However, it is also possible to determine whether a surface is curved based on the spread of the histogram. For example, a probability distribution may be fitted using maximum likelihood estimation, and an index representing the variance or spread of the distribution may be used. The threshold value may also be set arbitrarily.

[0138] <Tenth embodiment> Next, a tenth embodiment of the present invention will be described. Note that the configuration of the image capture device 100 in the tenth embodiment is the same as that shown in Fig. 1, and therefore a description thereof will be omitted.

[0139] FIG. 33 is a flowchart showing a method for removing a polarized light component in the tenth embodiment. In S1001, the polarization intensity mode calculation unit 107 determines whether or not there is reflection. If the polarization intensity histogram obtained in the polarization intensity histogram generation unit 106 has a polarization intensity peak equal to or greater than a predetermined threshold, it is determined that there is a reflective area, and the process proceeds to S1003. If not, it is determined that there is no reflective area, and the process proceeds to S1002, where the process ends without removing the polarization.

[0140] In S1003, it is determined whether or not there is an edge. If there is no edge, the process proceeds to S1004, where polarization is removed using the polarization removal method described in the first embodiment. On the other hand, if there is an edge, the process proceeds to S1005.

[0141] In S1005, it is determined whether the edge direction is known. If the edge direction is unknown, the process proceeds to S1006, and if the edge direction is known, the process proceeds to S1007.

[0142] In S1006, the pixel value of the smallest level among the four local pixels including the pixel of interest is used as the pixel value of the pixel of interest.

[0143] In S1007, it is determined whether the polarization intensity changes across the edge. If the polarization intensity does not change across the edge, the process proceeds to S1008, where polarization is removed using the polarization removal method described in the first embodiment. If the polarization intensity changes across the edge, the process proceeds to S1009.

[0144] In S1009, when the region is divided into two at the edge, polarization removal is performed by subtracting the polarization component of the pixel of interest using representative polarization characteristics calculated from multiple local four pixels that exist in the same region as the pixel of interest.

[0145] Fig. 34 shows pixels to be referenced in determining an edge in the tenth embodiment. The edge determination methods in S1003, S1005, and S1007 will be described with reference to Fig. 34.

[0146] In determining whether an edge exists in S1003, first, an approximation function F(θ) of the polarization characteristics is calculated using pixels P33, P34, P43, and P44 in the local four-pixel 1100 to be determined. This is defined as a first approximation function F1(θ). Next, an approximation function F(θ) of the polarization characteristics is calculated using pixels P23, P24, P53, and P54 above and below the local four-pixel 1100. This is defined as a second approximation function F2(θ). Furthermore, an approximation function F(θ) of the polarization characteristics is calculated using pixels P32, P42, P35, and P45 on the left and right of the local four-pixel 1100. This is defined as a third approximation function F3(θ). Furthermore, an approximation function F(θ) of the polarization characteristics is calculated using pixels P22, P25, P52, and P55 that are diagonally adjacent to the local four pixel 1100. This is defined as a fourth approximation function F4(θ).

[0147] Then, the difference between the polarization angle of the first approximation function F1(θ) and the polarization angle of the second approximation function F2(θ) is defined as the first difference. The difference between the polarization angle of the first approximation function F1(θ) and the polarization angle of the third approximation function F3(θ) is defined as the second difference.

[0148] If the first difference and the second difference are each less than a predetermined threshold, it is determined in S1003 that there is no edge, and if either the first difference or the second difference is equal to or greater than a predetermined threshold, it is determined in S1003 that there is an edge.

[0149] That is, if the first difference is equal to or greater than a predetermined threshold and the difference between the second differences is less than the predetermined threshold, it is determined in S1003 that there is an edge between pixels P33, P34, P43, and P44 within the local quad 1100. In this case, the direction of the edge is horizontal, so in S1005 it is determined that the direction of the edge can be determined.

[0150] Also, if the first difference is less than the predetermined threshold and the difference between the second differences is equal to or greater than the predetermined threshold, it is determined in S1003 that there is an edge between pixels P33, P34, P43, and P44 in the local 4-pixel group 1100. In this case, the edge direction is vertical, so it is determined in S1005 that the edge direction can be determined.

[0151] Furthermore, if both the first difference and the second difference are equal to or greater than a predetermined threshold, it is determined in S1003 that there is an edge between pixels P33, P34, P43, and P44 in the local 4-pixel group 1100, but in S1005 it is determined that the direction of the edge is unknown.

[0152] The method of determining the change in polarization intensity across the edge in S1007 will be described below.

[0153] The polarization intensity obtained from pixels P13, P14, P23, and P24 included in another local quad pixel above the local quad pixel 1100 is defined as a first polarization intensity.

[0154] Similarly, the polarization intensity obtained from pixels P31, P32, P41, and P42 included in another local quad to the left of the local quad 1100 is defined as a second polarization intensity.

[0155] Furthermore, the polarization intensity obtained from pixels P35, P36, P45, and P46 included in another local quad located to the right of the local quad 1100 is defined as a third polarization intensity.

[0156] Then, the polarization intensity obtained from pixels P53, P54, P63, and P64 included in another local quad below the local quad 1100 is set as a fourth polarization intensity.

[0157] In S1007, if there is an edge between pixels P33, P34, P43, and P44 in the local quad pixel 1100, it is determined that there is a change in polarization intensity across the edge if the difference between the first polarization intensity and the fourth polarization intensity is equal to or greater than a predetermined threshold, and it is determined that there is no change in polarization intensity across the edge if the difference between the first polarization intensity and the fourth polarization intensity is less than the predetermined threshold.

[0158] Furthermore, if there is an edge between pixels P33, P34, P43, and P44 in the local quad pixel 1100, and the difference between the second and third polarization intensities is equal to or greater than a predetermined threshold, it is determined that there is a change in polarization intensity across the edge.If the difference between the second and third polarization intensities is less than the predetermined threshold, it is determined that there is no change in polarization intensity across the edge.

[0159] As described above, in the tenth embodiment, by subtracting the polarization component calculated from the representative polarization characteristics from the brightness value of each pixel, it is possible to generate an image from which polarization has been removed (corrected) without reducing resolution. Furthermore, by determining edges using surrounding pixels and switching the polarization removal method, it is possible to appropriately remove polarization (correct) even from pixels that contain edges of both unpolarized and polarized components.

[0160] <Other embodiments> The present invention may be applied to a system consisting of multiple devices (e.g., a host computer, an interface device, a scanner, a video camera, etc.), or to an apparatus consisting of a single device (e.g., a copier, a facsimile machine, etc.).

[0161] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0162] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0163] 101, 701: imaging element, 102, 702: image acquisition unit, 103: area determination unit, 104: approximation function calculation unit, 105: polarization intensity extraction unit, 106: polarization intensity histogram generation unit, 107: polarization intensity mode calculation unit, 108: polarization angle extraction unit, 109: polarization angle histogram generation unit, 110: polarization angle mode calculation unit, 111: polarization removal processing unit, 501: gain conversion unit, 502: HDR synthesis unit, 703: image generation unit, 704: edge detection unit, 705: edge discrimination unit, 706: polarization characteristics processing unit

Claims

1. an input means for inputting an image signal generated by photoelectrically converting a plurality of polarized lights having different polarization angles; a first processing means for calculating, for each set of image signals based on the plurality of polarized light beams, a maximum value of polarization intensity of polarization components of the plurality of image signals and a polarization angle at which the polarization intensity is maximum; a setting means for determining a target area to be subjected to polarization removal based on the image signal, and for setting a statistical area included in the target area for taking statistics; a second processing means for determining a representative intensity, which is the most frequent value of the polarization intensities of the set included in the statistical domain, and a representative angle, which is the most frequent value of the polarization angles; a correction means for performing a first correction process that corrects each image signal of the set included in the target region according to a polarization angle of the each image signal, using the representative intensity and the representative angle; 1. An image processing device comprising:

2. The setting means Any region within the region in which the difference between the maximum value and the minimum value of the image signal in each set is equal to or greater than a predetermined first threshold is set as the statistical region.

2. The image processing device according to claim 1, wherein:

3. 3. The image processing device according to claim 1, wherein the second processing means generates a histogram of the polarization intensity and a histogram of the polarization angle of the set included in the statistical region, and calculates the mode of each.

4. The first processing means calculates an approximate function F(θ) for each set, where θ is the angle, A is the polarization intensity of the polarization component, B is the polarization angle of the polarization component, and C is the offset component. F(θ)=Acos(2θ+2B)+C 4. The image processing apparatus according to claim 1, wherein A is the polarization intensity and θ when F(θ) is at its maximum is the polarization angle.

5. The correction means calculates an approximate function F(θ) using the representative intensity as Imode, the representative angle as θmode, and C as an offset component. F(θ)=Imode・cos(2θ+2θmode)+C 5. The image processing apparatus according to claim 1, wherein the polarization component of each polarization angle is determined by calculating the polarization angle of the polarized light and substituting the polarization angle of the polarized light for θ.

6. further comprising a synthesizing means; the second processing means generates a histogram of the polarization intensities and a histogram of the polarization angles of the sets included in the statistical domain; When one peak exists in the histogram of the polarization intensity, the correction means performs the first correction process; performing a second correction process when a plurality of peaks exist in the histogram of polarization intensity; The second correction processing is a processing in which the setting means repeats a first processing of changing the size and position of the statistical region and a second processing of generating the histogram of the polarization intensity and the histogram of the polarization angle of the set included in the changed statistical region a predetermined number of times, the correction means performs correction using a representative intensity and a representative angle obtained based on the modes of the histogram of the polarization intensity and the histogram of the polarization angle generated by the second processing, and the synthesis means synthesizes the multiple corrected image signals obtained by the second processing.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. the second processing means generates a histogram of the polarization intensities and a histogram of the polarization angles of the sets included in the statistical domain; When one peak exists in the histogram of the polarization intensity, the second processing means calculates the representative intensity and the representative angle, and the correction means performs the first correction process; When a plurality of peaks exist in the histogram of polarization intensity, the second processing means does not obtain the representative intensity and the representative angle, and the correction means performs a third correction process to correct the image signal of the target area according to the polarization angle, using the polarization intensity and polarization angle of each set obtained by the first processing means.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

8. 8. The image processing device according to claim 6, wherein when the peak of the polarization intensity histogram cannot be determined, the second processing means regenerates the polarization intensity histogram using the polarization intensities of the set included in the statistical area that have the polarization angle within a predetermined range from the representative angle.

9. the correction means compares an intensity difference between the representative intensity and the polarization intensity of each set with a predetermined third threshold, and compares an angle difference between the representative angle and the polarization angle of each set with a predetermined fourth threshold; the correction means performs the first correction process on the set in which the intensity difference is less than a third threshold and the angle difference is less than a fourth threshold; 6. The image processing device according to claim 1, wherein the first processing means changes the combination of image signals constituting the set for which at least one of the intensity difference and the angle difference is equal to or greater than the third threshold or the fourth threshold.

10. 10. The image processing device according to claim 9, wherein the first processing means performs a first process to determine the polarization intensities and polarization angles of a new set in which the combination is changed for the set in which at least one of the intensity difference and the angle difference is equal to or greater than the third threshold or the fourth threshold, and the correction means performs a second process to compare the intensity difference and the angle difference for the new set with the third threshold and the fourth threshold, respectively, which are predetermined, until the intensity difference and the angle difference become less than the third threshold and the fourth threshold, respectively.

11. 11. The image processing device according to claim 9, wherein, when the image signals have color information, the first processing means performs processing to match the intensity of the new set of image signals resulting from the changed combination to the intensity of an image signal of a predetermined color, and calculates the polarization intensity and the polarization angle using the image signals after the processing.

12. detection means for detecting the presence or absence of an edge in each of the sets; a determining unit that determines the type of an edge when the detecting unit detects the edge, When the edge determined by the determining means is an edge of a non-polarized component present in the target area, the first correction process is performed, and when the edge is an edge of a polarized component present in the target area, the second correction process is performed.

7. The image processing device according to claim 6,

13. the second processing means generates a histogram of the polarization intensities and a histogram of the polarization angles of the sets included in the statistical domain; When there is one peak in the histogram of the polarization angles, the correction means performs the first correction process; 6. The image processing device according to claim 1, wherein, when a plurality of peaks exist in the histogram of polarization angles, the setting means performs a first process of reducing the size of the statistical region, the second processing means performs a second process of generating a histogram of the polarization intensity and a histogram of the polarization angle of the set included in the changed statistical region, and the first process and the second process are repeated until the histogram of the polarization angle generated in the second process has only one peak, and the correction means uses the representative intensity and representative angle of the changed statistical region to correct each image signal of the set included in the statistical region according to the polarization angle of each image signal.

14. the second processing means generates a histogram of the polarization intensities and a histogram of the polarization angles of the sets included in the statistical domain; When the shape of the histogram of the polarization angles satisfies a predetermined condition, the correction means performs the first correction process; 6. The image processing device according to claim 1, wherein, when the shape of the histogram of polarization angles does not satisfy the predetermined condition, the setting means performs a first process to reduce the size of the statistical region, the second processing means performs a second process to generate a histogram of the polarization intensity and a histogram of the polarization angle of the set included in the changed statistical region, and the first process and the second process are repeated until the shape of the histogram of polarization angles generated in the second process satisfies the condition, and the correction means uses the representative intensity and representative angle of the changed statistical region to correct each image signal of the set included in the statistical region according to the polarization angle of each image signal.

15. The image processing device according to claim 14, characterized in that the condition is that the variance of the histogram of polarization angles is less than a predetermined threshold, or that an index representing the spread of the variation or distribution obtained by fitting a probability distribution to the histogram of polarization angles by maximum likelihood estimation is smaller than a predetermined variation or distribution.

16. further comprising a detection means for detecting the presence or absence of an edge in each of the sets; The correction means performing the first correction process when no edge is detected by the detection means, or when an edge is detected and the polarization intensity does not change across the edge; If an edge is detected and the direction of the edge is unknown, the pixel signal to be corrected is replaced with the minimum value of the set including the pixel signal; When an edge is detected and the polarization intensity changes across the edge, the luminance value of the pixel signal to be corrected is corrected in accordance with the polarization angle of the pixel signal to be corrected, using the representative intensities and representative angles of the plurality of sets in the region to which the pixel signal belongs.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

17. 17. The image processing apparatus according to claim 16, wherein said detecting means detects the presence or absence of an edge based on the polarization intensity of an approximation function of polarization characteristics calculated from peripheral pixels of said set.

18. 18. The image processing device according to claim 16, wherein said detecting means detects the edge direction based on a polarization angle of an approximation function of polarization characteristics calculated from peripheral pixels of said set.

19. an imaging element including a plurality of polarizing filters that transmit a plurality of polarized light beams having different polarization angles; An image processing device according to any one of claims 1 to 18, An imaging device comprising:

20. 20. The imaging device according to claim 19, wherein the imaging element is further covered with color filters of a plurality of colors, and color filters of the same colors are arranged for each of a plurality of pixels corresponding to the set.

21. performing exposure using a plurality of different exposure conditions on a plurality of pixels corresponding to each set; the image processing device includes: a conversion means for adjusting the image signal in accordance with an exposure condition of the image signal for each pixel input from the input means; and a synthesis means for performing synthesis processing to expand the dynamic range.

21. The imaging device according to claim 19, wherein the image processing device processes the adjusted image signal, and the combining means performs the combining process using the image signal from which the polarization component has been removed.

22. an input step in which an input means inputs image signals generated by photoelectrically converting a plurality of polarized light beams having different polarization angles; a first processing step in which a first processing means, for each set of image signals based on the plurality of polarized light beams, determines a maximum value of polarization intensity of polarization components of the plurality of image signals and a polarization angle at which the polarization intensity is maximized; a setting step in which a setting means determines a target area to be subjected to polarization removal based on the image signal, and sets a statistical area included in the target area for taking statistics; a second processing step in which a second processing means determines a representative intensity, which is the most frequent value of the polarization intensities of the set included in the statistical region, and a representative angle, which is the most frequent value of the polarization angles; a correction step in which a correction means corrects each image signal of the set included in the target region according to the polarization angle of each image signal using the representative intensity and the representative angle; An image processing method comprising:

23. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 18.

24. A computer-readable storage medium storing the program according to claim 23.

Images